Advanced Calculus with Applications in Statistics (Wiley Series in Probability and Statistics)

Advanced Calculus with Applications in Statistics Second Edition Revised and Expanded

Andre ´ I. Khuri University of Florida Gainesville, Florida

Advanced Calculus with Applications in Statistics Second Edition

Advanced Calculus with Applications in Statistics Second Edition Revised and Expanded

Andre ´ I. Khuri University of Florida Gainesville, Florida

Copyright 䊚 2003 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, Ž978. 750-8400, fax Ž978. 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, Ž201. 748-6011, fax Ž201. 748-6008, e-mail: [email protected]. Limit of LiabilityrDisclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic format. Library of Congress Cataloging-in-Publication Data Khuri, Andre ´ I., 1940Advanced calculus with applications in statistics r Andre ´ I. Khuri. -- 2nd ed. rev. and expended. p. cm. -- ŽWiley series in probability and statistics . Includes bibliographical references and index. ISBN 0-471-39104-2 Žcloth : alk. paper. 1. Calculus. 2. Mathematical statistics. I. Title. II. Series. QA303.2.K48 2003 515--dc21 Printed in the United States of America 10 9 8 7 6 5 4 3 2 1

2002068986

To Ronnie, Marcus, and Roxanne and In memory of my sister Ninette

Contents

Preface

xv

Preface to the First Edition 1.

An Introduction to Set Theory

xvii 1

1.1. The Concept of a Set, 1 1.2. Set Operations, 2 1.3. Relations and Functions, 4 1.4. Finite, Countable, and Uncountable Sets, 6 1.5. Bounded Sets, 9 1.6. Some Basic Topological Concepts, 10 1.7. Examples in Probability and Statistics, 13 Further Reading and Annotated Bibliography, 15 Exercises, 17 2.

Basic Concepts in Linear Algebra 2.1. 2.2. 2.3.

21

Vector Spaces and Subspaces, 21 Linear Transformations, 25 Matrices and Determinants, 27 2.3.1. Basic Operations on Matrices, 28 2.3.2. The Rank of a Matrix, 33 2.3.3. The Inverse of a Matrix, 34 2.3.4. Generalized Inverse of a Matrix, 36 2.3.5. Eigenvalues and Eigenvectors of a Matrix, 36 2.3.6. Some Special Matrices, 38 2.3.7. The Diagonalization of a Matrix, 38 2.3.8. Quadratic Forms, 39 vii

viii

CONTENTS

2.3.9.

The Simultaneous Diagonalization of Matrices, 40 2.3.10. Bounds on Eigenvalues, 41 2.4. Applications of Matrices in Statistics, 43 2.4.1. The Analysis of the Balanced Mixed Model, 43 2.4.2. The Singular-Value Decomposition, 45 2.4.3. Extrema of Quadratic Forms, 48 2.4.4. The Parameterization of Orthogonal Matrices, 49 Further Reading and Annotated Bibliography, 50 Exercises, 53 3.

Limits and Continuity of Functions

57

3.1. 3.2. 3.3. 3.4.

Limits of a Function, 57 Some Properties Associated with Limits of Functions, 63 The o, O Notation, 65 Continuous Functions, 66 3.4.1. Some Properties of Continuous Functions, 71 3.4.2. Lipschitz Continuous Functions, 75 3.5. Inverse Functions, 76 3.6. Convex Functions, 79 3.7. Continuous and Convex Functions in Statistics, 82 Further Reading and Annotated Bibliography, 87 Exercises, 88 4.

Differentiation 4.1. 4.2. 4.3. 4.4.

The Derivative of a Function, 93 The Mean Value Theorem, 99 Taylor’s Theorem, 108 Maxima and Minima of a Function, 112 4.4.1. A Sufficient Condition for a Local Optimum, 114 4.5. Applications in Statistics, 115 4.5.1. Functions of Random Variables, 116 4.5.2. Approximating Response Functions, 121 4.5.3. The Poisson Process, 122 4.5.4. Minimizing the Sum of Absolute Deviations, 124 Further Reading and Annotated Bibliography, 125 Exercises, 127

93

ix

CONTENTS

5.

Infinite Sequences and Series 5.1.

Infinite Sequences, 132 5.1.1.

5.2.

The Cauchy Criterion, 137

Infinite Series, 140 5.2.1. 5.2.2. 5.2.3. 5.2.4.

5.3.

132

Tests of Convergence for Series of Positive Terms, 144 Series of Positive and Negative Terms, 158 Rearrangement of Series, 159 Multiplication of Series, 162

Sequences and Series of Functions, 165 5.3.1.

Properties of Uniformly Convergent Sequences and Series, 169

5.4.

Power Series, 174

5.5.

Sequences and Series of Matrices, 178

5.6.

Applications in Statistics, 182 5.6.1. 5.6.2. 5.6.3.

5.6.4. 5.6.5. 5.6.6.

Moments of a Discrete Distribution, 182 Moment and Probability Generating Functions, 186 Some Limit Theorems, 191 5.6.3.1. The Weak Law of Large Numbers ŽKhinchine’s Theorem., 192 5.6.3.2. The Strong Law of Large Numbers ŽKolmogorov’s Theorem., 192 5.6.3.3. The Continuity Theorem for Probability Generating Functions, 192 Power Series and Logarithmic Series Distributions, 193 Poisson Approximation to Power Series Distributions, 194 A Ridge Regression Application, 195

Further Reading and Annotated Bibliography, 197 Exercises, 199 6.

Integration

205

6.1.

Some Basic Definitions, 205

6.2.

The Existence of the Riemann Integral, 206

6.3.

Some Classes of Functions That Are Riemann Integrable, 210 6.3.1.

Functions of Bounded Variation, 212

x

CONTENTS

6.4.

Properties of the Riemann Integral, 215 6.4.1. Change of Variables in Riemann Integration, 219 6.5. Improper Riemann Integrals, 220 6.5.1. Improper Riemann Integrals of the Second Kind, 225 6.6. Convergence of a Sequence of Riemann Integrals, 227 6.7. Some Fundamental Inequalities, 229 6.7.1. The Cauchy᎐Schwarz Inequality, 229 6.7.2. Holder’s Inequality, 230 ¨ 6.7.3. Minkowski’s Inequality, 232 6.7.4. Jensen’s Inequality, 233 6.8. Riemann᎐Stieltjes Integral, 234 6.9. Applications in Statistics, 239 6.9.1. The Existence of the First Negative Moment of a Continuous Distribution, 242 6.9.2. Transformation of Continuous Random Variables, 246 6.9.3. The Riemann᎐Stieltjes Representation of the Expected Value, 249 6.9.4. Chebyshev’s Inequality, 251 Further Reading and Annotated Bibliography, 252 Exercises, 253 7.

Multidimensional Calculus 7.1. 7.2. 7.3. 7.4.

7.5. 7.6. 7.7. 7.8. 7.9.

Some Basic Definitions, 261 Limits of a Multivariable Function, 262 Continuity of a Multivariable Function, 264 Derivatives of a Multivariable Function, 267 7.4.1. The Total Derivative, 270 7.4.2. Directional Derivatives, 273 7.4.3. Differentiation of Composite Functions, 276 Taylor’s Theorem for a Multivariable Function, 277 Inverse and Implicit Function Theorems, 280 Optima of a Multivariable Function, 283 The Method of Lagrange Multipliers, 288 The Riemann Integral of a Multivariable Function, 293 7.9.1. The Riemann Integral on Cells, 294 7.9.2. Iterated Riemann Integrals on Cells, 295 7.9.3. Integration over General Sets, 297 7.9.4. Change of Variables in n-Tuple Riemann Integrals, 299

261

xi

CONTENTS

7.10. 7.11.

Differentiation under the Integral Sign, 301 Applications in Statistics, 304 7.11.1. Transformations of Random Vectors, 305 7.11.2. Maximum Likelihood Estimation, 308 7.11.3. Comparison of Two Unbiased Estimators, 310 7.11.4. Best Linear Unbiased Estimation, 311 7.11.5. Optimal Choice of Sample Sizes in Stratified Sampling, 313 Further Reading and Annotated Bibliography, 315 Exercises, 316

8.

Optimization in Statistics 8.1.

8.2.

8.3.

8.4.

8.5. 8.6. 8.7. 8.8.

8.9.

The Gradient Methods, 329 8.1.1. The Method of Steepest Descent, 329 8.1.2. The Newton᎐Raphson Method, 331 8.1.3. The Davidon᎐Fletcher ᎐Powell Method, 331 The Direct Search Methods, 332 8.2.1. The Nelder᎐Mead Simplex Method, 332 8.2.2. Price’s Controlled Random Search Procedure, 336 8.2.3. The Generalized Simulated Annealing Method, 338 Optimization Techniques in Response Surface Methodology, 339 8.3.1. The Method of Steepest Ascent, 340 8.3.2. The Method of Ridge Analysis, 343 8.3.3. Modified Ridge Analysis, 350 Response Surface Designs, 355 8.4.1. First-Order Designs, 356 8.4.2. Second-Order Designs, 358 8.4.3. Variance and Bias Design Criteria, 359 Alphabetic Optimality of Designs, 362 Designs for Nonlinear Models, 367 Multiresponse Optimization, 370 Maximum Likelihood Estimation and the EM Algorithm, 372 8.8.1. The EM Algorithm, 375 Minimum Norm Quadratic Unbiased Estimation of Variance Components, 378

327

xii

CONTENTS

Scheffe’s ´ Confidence Intervals, 382 8.10.1. The Relation of Scheffe’s ´ Confidence Intervals to the F-Test, 385 Further Reading and Annotated Bibliography, 391 Exercises, 395 8.10.

9.

Approximation of Functions

403

9.1. 9.2.

Weierstrass Approximation, 403 Approximation by Polynomial Interpolation, 410 9.2.1. The Accuracy of Lagrange Interpolation, 413 9.2.2. A Combination of Interpolation and Approximation, 417 9.3. Approximation by Spline Functions, 418 9.3.1. Properties of Spline Functions, 418 9.3.2. Error Bounds for Spline Approximation, 421 9.4. Applications in Statistics, 422 9.4.1. Approximate Linearization of Nonlinear Models by Lagrange Interpolation, 422 9.4.2. Splines in Statistics, 428 9.4.2.1. The Use of Cubic Splines in Regression, 428 9.4.2.2. Designs for Fitting Spline Models, 430 9.4.2.3. Other Applications of Splines in Statistics, 431 Further Reading and Annotated Bibliography, 432 Exercises, 434 10.

Orthogonal Polynomials 10.1. 10.2.

10.3. 10.4.

10.5. 10.6. 10.7.

Introduction, 437 Legendre Polynomials, 440 10.2.1. Expansion of a Function Using Legendre Polynomials, 442 Jacobi Polynomials, 443 Chebyshev Polynomials, 444 10.4.1. Chebyshev Polynomials of the First Kind, 444 10.4.2. Chebyshev Polynomials of the Second Kind, 445 Hermite Polynomials, 447 Laguerre Polynomials, 451 Least-Squares Approximation with Orthogonal Polynomials, 453

437

xiii

CONTENTS

10.8. 10.9.

Orthogonal Polynomials Defined on a Finite Set, 455 Applications in Statistics, 456 10.9.1. Applications of Hermite Polynomials, 456 10.9.1.1. Approximation of Density Functions and Quantiles of Distributions, 456 10.9.1.2. Approximation of a Normal Integral, 460 10.9.1.3. Estimation of Unknown Densities, 461 10.9.2. Applications of Jacobi and Laguerre Polynomials, 462 10.9.3. Calculation of Hypergeometric Probabilities Using Discrete Chebyshev Polynomials, 462 Further Reading and Annotated Bibliography, 464 Exercises, 466 11.

Fourier Series

471

11.1. 11.2. 11.3. 11.4. 11.5.

Introduction, 471 Convergence of Fourier Series, 475 Differentiation and Integration of Fourier Series, 483 The Fourier Integral, 488 Approximation of Functions by Trigonometric Polynomials, 495 11.5.1. Parseval’s Theorem, 496 11.6. The Fourier Transform, 497 11.6.1. Fourier Transform of a Convolution, 499 11.7. Applications in Statistics, 500 11.7.1. Applications in Time Series, 500 11.7.2. Representation of Probability Distributions, 501 11.7.3. Regression Modeling, 504 11.7.4. The Characteristic Function, 505 11.7.4.1. Some Properties of Characteristic Functions, 510 Further Reading and Annotated Bibliography, 510 Exercises, 512 12.

Approximation of Integrals 12.1. 12.2. 12.3.

The Trapezoidal Method, 517 12.1.1. Accuracy of the Approximation, 518 Simpson’s Method, 521 Newton᎐Cotes Methods, 523

517

xiv

CONTENTS

12.4. 12.5. 12.6. 12.7. 12.8.

Gaussian Quadrature, 524 Approximation over an Infinite Interval, 528 The Method of Laplace, 531 Multiple Integrals, 533 The Monte Carlo Method, 535 12.8.1. Variation Reduction, 537 12.8.2. Integrals in Higher Dimensions, 540 12.9. Applications in Statistics, 541 12.9.1. The Gauss᎐Hermite Quadrature, 542 12.9.2. Minimum Mean Squared Error Quadrature, 543 12.9.3. Moments of a Ratio of Quadratic Forms, 546 12.9.4. Laplace’s Approximation in Bayesian Statistics, 548 12.9.5. Other Methods of Approximating Integrals in Statistics, 549 Further Reading and Annotated Bibliography, 550 Exercises, 552 Appendix. Solutions to Selected Exercises Chapter Chapter Chapter Chapter Chapter Chapter Chapter Chapter Chapter Chapter Chapter Chapter

557

1, 557 2, 560 3, 565 4, 570 5, 577 6, 590 7, 600 8, 613 9, 622 10, 627 11, 635 12, 644

General Bibliography

652

Index

665

Preface

This edition provides a rather substantial addition to the material covered in the first edition. The principal difference is the inclusion of three new chapters, Chapters 10, 11, and 12, in addition to an appendix of solutions to exercises. Chapter 10 covers orthogonal polynomials, such as Legendre, Chebyshev, Jacobi, Laguerre, and Hermite polynomials, and discusses their applications in statistics. Chapter 11 provides a thorough coverage of Fourier series. The presentation is done in such a way that a reader with no prior knowledge of Fourier series can have a clear understanding of the theory underlying the subject. Several applications of Fouries series in statistics are presented. Chapter 12 deals with approximation of Riemann integrals. It gives an exposition of methods for approximating integrals, including those that are multidimensional. Applications of some of these methods in statistics are discussed. This subject area has recently gained prominence in several fields of science and engineering, and, in particular, Bayesian statistics. The material should be helpful to readers who may be interested in pursuing further studies in this area. A significant addition is the inclusion of a major appendix that gives detailed solutions to the vast majority of the exercises in Chapters 1᎐12. This supplement was prepared in response to numerous suggestions by users of the first edition. The solutions should also be helpful in getting a better understanding of the various topics covered in the book. In addition to the aforementioned material, several new exercises were added to some of the chapters in the first edition. Chapter 1 was expanded by the inclusion of some basic topological concepts. Chapter 9 was modified to accommodate Chapter 10. The changes in the remaining chapters, 2 through 8, are very minor. The general bibliography was updated. The choice of the new chapters was motivated by the evolution of the field of statistics and the growing needs of statisticians for mathematical tools beyond the realm of advanced calculus. This is certainly true in topics concerning approximation of integrals and distribution functions, stochastic xv

xvi

PREFACE

processes, time series analysis, and the modeling of periodic response functions, to name just a few. The book is self-contained. It can be used as a text for a two-semester course in advanced calculus and introductory mathematical analysis. Chapters 1᎐7 may be covered in one semester, and Chapters 8᎐12 in the other semester. With its coverage of a wide variety of topics, the book can also serve as a reference for statisticians, and others, who need an adequate knowledge of mathematics, but do not have the time to wade through the myriad mathematics books. It is hoped that the inclusion of a separate section on applications in statistics in every chapter will provide a good motivation for learning the material in the book. This represents a continuation of the practice followed in the first edition. As with the first edition, the book is intended as much for mathematicians as for statisticians. It can easily be turned into a pure mathematics book by simply omitting the section on applications in statistics in a given chapter. Mathematicians, however, may find the sections on applications in statistics to be quite useful, particularly to mathematics students seeking an interdisciplinary major. Such a major is becoming increasingly popular in many circles. In addition, several topics are included here that are not usually found in a typical advanced calculus book, such as approximation of functions and integrals, Fourier series, and orthogonal polynomials. The fields of mathematics and statistics are becoming increasingly intertwined, making any separation of the two unpropitious. The book represents a manifestation of the interdependence of the two fields. The mathematics background needed for this edition is the same as for the first edition. For readers interested in statistical applications, a background in introductory mathematical statistics will be helpful, but not absolutely essential. The annotated bibliography in each chapter can be consulted for additional readings. I am grateful to all those who provided comments and helpful suggestions concerning the first edition, and to my wife Ronnie for her help and support. ANDRE ´ I. KHURI Gaines®ille, Florida

Preface to the First Edition

The most remarkable mathematical achievement of the seventeenth century was the invention of calculus by Isaac Newton Ž1642᎐1727. and Gottfried Wilhelm Leibniz Ž1646᎐1716.. It has since played a significant role in all fields of science, serving as its principal quantitative language. There is hardly any scientific discipline that does not require a good knowledge of calculus. The field of statistics is no exception. Advanced calculus has had a fundamental and seminal role in the development of the basic theory underlying statistical methodology. With the rapid growth of statistics as a discipline, particularly in the last three decades, knowledge of advanced calculus has become imperative for understanding the recent advances in this field. Students as well as research workers in statistics are expected to have a certain level of mathematical sophistication in order to cope with the intricacies necessitated by the emerging of new statistical methodologies. This book has two purposes. The first is to provide beginning graduate students in statistics with the basic concepts of advanced calculus. A high percentage of these students have undergraduate training in disciplines other than mathematics with only two or three introductory calculus courses. They are, in general, not adequately prepared to pursue an advanced graduate degree in statistics. This book is designed to fill the gaps in their mathematical training and equip them with the advanced calculus tools needed in their graduate work. It can also provide the basic prerequisites for more advanced courses in mathematics. One salient feature of this book is the inclusion of a complete section in each chapter describing applications in statistics of the material given in the chapter. Furthermore, a large segment of Chapter 8 is devoted to the important problem of optimization in statistics. The purpose of these applications is to help motivate the learning of advanced calculus by showing its relevance in the field of statistics. There are many advanced calculus books designed for engineers or business majors, but there are none for statistics xvii

xviii

PREFACE TO THE FIRST EDITION

majors. This is the first advanced calculus book to emphasize applications in statistics. The scope of this book is not limited to serving the needs of statistics graduate students. Practicing statisticians can use it to sharpen their mathematical skills, or they may want to keep it as a handy reference for their research work. These individuals may be interested in the last three chapters, particularly Chapters 8 and 9, which include a large number of citations of statistical papers. The second purpose of the book concerns mathematics majors. The book’s thorough and rigorous coverage of advanced calculus makes it quite suitable as a text for juniors or seniors. Chapters 1 through 7 can be used for this purpose. The instructor may choose to omit the last section in each chapter, which pertains to statistical applications. Students may benefit, however, from the exposure to these additional applications. This is particularly true given that the trend today is to allow the undergraduate student to have a major in mathematics with a minor in some other discipline. In this respect, the book can be particularly useful to those mathematics students who may be interested in a minor in statistics. Other features of this book include a detailed coverage of optimization techniques and their applications in statistics ŽChapter 8., and an introduction to approximation theory ŽChapter 9.. In addition, an annotated bibliography is given at the end of each chapter. This bibliography can help direct the interested reader to other sources in mathematics and statistics that are relevant to the material in a given chapter. A general bibliography is provided at the end of the book. There are also many examples and exercises in mathematics and statistics in every chapter. The exercises are classified by discipline Žmathematics and statistics . for the benefit of the student and the instructor. The reader is assumed to have a mathematical background that is usually obtained in the freshman᎐sophomore calculus sequence. A prerequisite for understanding the statistical applications in the book is an introductory statistics course. Obviously, those not interested in such applications need not worry about this prerequisite. Readers who do not have any background in statistics, but are nevertheless interested in the application sections, can make use of the annotated bibliography in each chapter for additional reading. The book contains nine chapters. Chapters 1᎐7 cover the main topics in advanced calculus, while chapters 8 and 9 include more specialized subject areas. More specifically, Chapter 1 introduces the basic elements of set theory. Chapter 2 presents some fundamental concepts concerning vector spaces and matrix algebra. The purpose of this chapter is to facilitate the understanding of the material in the remaining chapters, particularly, in Chapters 7 and 8. Chapter 3 discusses the concepts of limits and continuity of functions. The notion of differentiation is studied in Chapter 4. Chapter 5 covers the theory of infinite sequences and series. Integration of functions is

PREFACE TO THE FIRST EDITION

xix

the theme of Chapter 6. Multidimensional calculus is introduced in Chapter 7. This chapter provides an extension of the concepts of limits, continuity, differentiation, and integration to functions of several variables Žmultivariable functions.. Chapter 8 consists of two parts. The first part presents an overview of the various methods of optimization of multivariable functions whose optima cannot be obtained explicitly by standard advanced calculus techniques. The second part discusses a variety of topics of interest to statisticians. The common theme among these topics is optimization. Finally, Chapter 9 deals with the problem of approximation of continuous functions with polynomial and spline functions. This chapter is of interest to both mathematicians and statisticians and contains a wide variety of applications in statistics. I am grateful to the University of Florida for granting me a sabbatical leave that made it possible for me to embark on the project of writing this book. I would also like to thank Professor Rocco Ballerini at the University of Florida for providing me with some of the exercises used in Chapters, 3, 4, 5, and 6. ANDRE ´ I. KHURI Gaines®ille, Florida

CHAPTER 1

An Introduction to Set Theory

The origin of the modern theory of sets can be traced back to the Russian-born German mathematician Georg Cantor Ž1845᎐1918.. This chapter introduces the basic elements of this theory. 1.1. THE CONCEPT OF A SET A set is any collection of well-defined and distinguishable objects. These objects are called the elements, or members, of the set and are denoted by lowercase letters. Thus a set can be perceived as a collection of elements united into a single entity. Georg Cantor stressed this in the following words: ‘‘A set is a multitude conceived of by us as a one.’’ If x is an element of a set A, then this fact is denoted by writing x g A. If, however, x is not an element of A, then we write x f A. Curly brackets are usually used to describe the contents of a set. For example, if a set A consists of the elements x 1 , x 2 , . . . , x n , then it can be represented as A s
x 1 , x 2 , . . . , x n4 . In the event membership in a set is determined by the satisfaction of a certain property or a relationship, then the description of the same can be given within the curly brackets. For example, if A consists of all real numbers x such that x 2 ) 1, then it can be expressed as A s
x < x 2 ) 14 , where the bar < is used simply to mean ‘‘such that.’’ The definition of sets in this manner is based on the axiom of abstraction, which states that given any property, there exists a set whose elements are just those entities having that property. Definition 1.1.1. The set that contains no elements is called the empty set and is denoted by ⭋. I Definition 1.1.2. A set A is a subset of another set B, written symbolically as A ; B, if every element of A is an element of B. If B contains at least one element that is not in A, then A is said to be a proper subset of B. I 1

2

AN INTRODUCTION TO SET THEORY

Definition 1.1.3. A set A and a set B are equal if A ; B and B ; A. Thus, every element of A is an element of B and vice versa. I Definition 1.1.4. The set that contains all sets under consideration in a certain study is called the universal set and is denoted by ⍀. I

1.2. SET OPERATIONS There are two basic operations for sets that produce new sets from existing ones. They are the operations of union and intersection. Definition 1.2.1. The union of two sets A and B, denoted by A j B, is the set of elements that belong to either A or B, that is, A j B s
x < x g A or xg B 4 .

I

This definition can be extended to more than two sets. For example, if n A1 , A 2 , . . . , A n are n given sets, then their union, denoted by D is1 A i , is a set such that x is an element of it if and only if x belongs to at least one of the A i Ž i s 1, 2, . . . , n.. Definition 1.2.2. The intersection of two sets A and B, denoted by A l B, is the set of elements that belong to both A and B. Thus A l B s
x < x g A and xg B 4 .

I

This definition can also be extended to more than two sets. As before, if n A1 , A 2 , . . . , A n are n given sets, then their intersection, denoted by F is1 Ai, Ž is the set consisting of all elements that belong to all the A i i s 1, 2, . . . , n.. Definition 1.2.3. Two sets A and B are disjoint if their intersection is the empty set, that is, A l B s ⭋. I Definition 1.2.4. The complement of a set A, denoted by A, is the set consisting of all elements in the universal set that do not belong to A. In other words, xg A if and only if xf A. The complement of A with respect to a set B is the set B y A which consists of the elements of B that do not belong to A. This complement is called the relative complement of A with respect to B. I From Definitions 1.1.1᎐1.1.4 and 1.2.1᎐1.2.4, the following results can be concluded: RESULT 1.2.1. The empty set ⭋ is a subset of every set. To show this, suppose that A is any set. If it is false that ⭋ ; A, then there must be an

3

SET OPERATIONS

element in ⭋ which is not in A. But this is not possible, since ⭋ is empty. It is therefore true that ⭋ ; A. RESULT 1.2.2. The empty set ⭋ is unique. To prove this, suppose that ⭋1 and ⭋2 are two empty sets. Then, by the previous result, ⭋1 ; ⭋2 and ⭋2 G ⭋1. Hence, ⭋1 s ⭋2 . RESULT 1.2.3. of ⍀ is ⭋.

The complement of ⭋ is ⍀. Vice versa, the complement

RESULT 1.2.4.

The complement of A is A.

RESULT 1.2.5.

For any set A, A j A s ⍀ and A l A s ⭋.

RESULT 1.2.6.

A y B s A y A l B.

RESULT 1.2.7.

A j Ž B j C . s Ž A j B . j C.

RESULT 1.2.8.

A l Ž B l C . s Ž A l B . l C.

RESULT 1.2.9.

A j Ž B l C . s Ž A j B . l Ž A j C ..

RESULT 1.2.10.

A l Ž B j C . s Ž A l B . j Ž A l C ..

RESULT 1.2.11.

n n A is F is1 Ai. Ž A j B . s A l B. More generally, D is1

RESULT 1.2.12.

n n A is D is1 Ai. Ž A l B . s A j B. More generally, F is1

Definition 1.2.5. Let A and B be two sets. Their Cartesian product, denoted by A = B, is the set of all ordered pairs Ž a, b . such that ag A and bg B, that is, A = B s
Ž a, b . < ag A and b g B 4 . The word ‘‘ordered’’ means that if a and c are elements in A and b and d are elements in B, then Ž a, b . s Ž c, d . if and only if as c and bs d. I The preceding definition can be extended to more than two sets. For example, if A1 , A 2 , . . . , A n are n given sets, then their Cartesian product is n denoted by =is1 A i and defined by n

= A s
Ž a , a , . . . , a . a g A , i s 1, 2, . . . , n4 . is1

i

1

2

n

i

i

4

AN INTRODUCTION TO SET THEORY

Here, Ž a1 , a2 , . . . , a n ., called an ordered n-tuple, represents a generalization of the ordered pair. In particular, if the A i are equal to A for n i s 1, 2, . . . , n, then one writes An for =is1 A. The following results can be easily verified: RESULT 1.2.13.

A = B s ⭋ if and only if A s ⭋ or B s ⭋.

RESULT 1.2.14.

Ž A j B . = C s Ž A = C . j Ž B = C ..

RESULT 1.2.15.

Ž A l B . = C s Ž A = C . l Ž B = C ..

RESULT 1.2.16.

Ž A = B . l Ž C = D . s Ž A l C . = Ž B l D ..

1.3. RELATIONS AND FUNCTIONS Let A = B be the Cartesian product of two sets, A and B. Definition 1.3.1. A relations ␳ from A to B is a subset of A = B, that is, ␳ consists of ordered pairs Ž a, b . such that ag A and bg B. In particular, if A s B, then ␳ is said to be a relation in A. For example, if A s
7, 8, 94 and B s
7, 8, 9, 104 , then ␳ s
Ž a, b .< a- b, ag A, bg B4 is a relation from A to B that consists of the six ordered pairs Ž7, 8., Ž7, 9., Ž7, 10., Ž8, 9., Ž8, 10., and Ž9, 10.. Whenever ␳ is a relation and Ž x, y . g ␳ , then x and y are said to be ␳-related. This is denoted by writing x ␳ y. I Definition 1.3.2. A relation ␳ in a set A is an equivalence relation if the following properties are satisfied: 1. ␳ is reflexive, that is, a ␳ a for any a in A. 2. ␳ is symmetric, that is, if a ␳ b, then b␳ a for any a, b in A. 3. ␳ is transitive, that is, if a ␳ b and b␳ c, then a ␳ c for any a, b, c in A. If ␳ is an equivalence relation in a set A, then for a given a0 in A, the set C Ž a0 . s
ag A < a0 ␳ a4 , which consists of all elements of A that are ␳-related to a0 , is called an equivalence class of a0 . I RESULT 1.3.1. ag C Ž a. for any a in A. Thus each element of A is an element of an equivalence class.

RELATIONS AND FUNCTIONS

5

RESULT 1.3.2. If C Ž a1 . and C Ž a2 . are two equivalence classes, then either C Ž a1 . s C Ž a2 ., or C Ž a1 . and C Ž a2 . are disjoint subsets. It follows from Results 1.3.1 and 1.3.2 that if A is a nonempty set, the collection of distinct ␳-equivalence classes of A forms a partition of A. As an example of an equivalence relation, consider that a ␳ b if and only if a and b are integers such that ay b is divisible by a nonzero integer n. This is the relation of congruence modulo n in the set of integers and is written symbolically as a' b Žmod n.. Clearly, a' a Žmod n., since ay as 0 is divisible by n. Also, if a' b Žmod n., then b' a Žmod n., since if ay b is divisible by n, then so is by a. Furthermore, if a' b Žmod n. and b' c Žmod n., then a' c Žmod n.. This is true because if ay b and by c are both divisible by n, then so is Ž ay b . q Ž by c . s ay c. Now, if a0 is a given integer, then a ␳-equivalence class of a0 consists of all integers that can be written as as a0 q kn, where k is an integer. This in this example C Ž a0 . is the set
a0 q kn < k g J 4 , where J denotes the set of all integers. Definition 1.3.3. Let ␳ be a relation from A to B. Suppose that ␳ has the property that for all x in A, if x ␳ y and x ␳ z, where y and z are elements in B, then y s z. Such a relation is called a function. I Thus a function is a relation ␳ such that any two elements in B that are ␳-related to the same x in A must be identical. In other words, to each element x in A, there corresponds only one element y in B. We call y the value of the function at x and denote it by writing y s f Ž x .. The set A is called the domain of the function f, and the set of all values of f Ž x . for x in A is called the range of f, or the image of A under f, and is denoted by f Ž A.. In this case, we say that f is a function, or a mapping, from A into B. We express this fact by writing f : A ™ B. Note that f Ž A. is a subset of B. In particular, if B s f Ž A., then f is said to be a function from A onto B. In this case, every element b in B has a corresponding element a in A such that bs f Ž a.. Definition 1.3.4. A function f defined on a set A is said to be a one-to-one function if whenever f Ž x 1 . s f Ž x 2 . for x 1 , x 2 in A, one has x 1 s x 2 . Equivalently, f is a one-to-one function if whenever x 1 / x 2 , one has f Ž x 1 . / f Ž x 2 .. I Thus a function f : A ™ B is one-to-one if to each y in f Ž A., there corresponds only one element x in A such that y s f Ž x .. In particular, if f is a one-to-one and onto function, then it is said to provide a one-to-one correspondence between A and B. In this case, the sets A and B are said to be equivalent. This fact is denoted by writing A ; B. Note that whenever A ; B, there is a function g: B ™ A such that if y s f Ž x ., then xs g Ž y .. The function g is called the inverse function of f and

6

AN INTRODUCTION TO SET THEORY

is denoted by fy1. It is easy to see that A ; B defines an equivalence relation. Properties 1 and 2 in Definition 1.3.2 are obviously true here. As for property 3, if A, B, and C are sets such that A ; B and B ; C, then A ; C. To show this, let f : A ™ B and h: B ™ C be one-to-one and onto functions. Then, the composite function h( f, where h( f Ž x . s hw f Ž x .x, defines a oneto-one correspondence between A and C. EXAMPLE 1.3.1. The relation a ␳ b, where a and b are real numbers such that as b 2 , is not a function. This is true because both pairs Ž a, b . and Ž a,y b . belong to ␳ . EXAMPLE 1.3.2. The relation a ␳ b, where a and b are real numbers such that bs 2 a2 q 1, is a function, since for each a, there is only one b that is ␳-related to a. EXAMPLE 1.3.3. Let A s
x < y1 F x F 14 , B s
x < 0 F x F 24 . Define f : A ™ B such that f Ž x . s x 2 . Here, f is a function, but is not one-to-one because f Ž1. s f Žy1. s 1. Also, f does not map A onto B, since y s 2 has no corresponding x in A such that x 2 s 2. EXAMPLE 1.3.4. Consider the relation x ␳ y, where y s arcsin x, y1 F x F 1. Here, y is an angle measured in radians whose sine is x. Since there are infinitely many angles with the same sine, ␳ is not a function. However, if we restrict the range of y to the set B s
y < y␲r2 F y F ␲r24 , then ␳ becomes a function, which is also one-to-one and onto. This function is the inverse of the sine function xs sin y. We refer to the values of y that belong to the set B as the principal values of arcsin x, which we denote by writing y s Arcsin x. Note that other functions could have also been defined from the arcsine relation. For example, if ␲r2 F y F 3␲r2, then xs sin y s ysin z, where z s y y ␲ . Since y␲r2F zF ␲r2, then z s yArcsin x. Thus y s ␲ y Arcsin x maps the set A s
x < y1 F x F 14 in a one-to-one manner onto the set C s
y < ␲r2 F y F 3␲r24 . 1.4. FINITE, COUNTABLE, AND UNCOUNTABLE SETS Let Jn s
1, 2, . . . , n4 be a set consisting of the first n positive integers, and let Jq denote the set of all positive integers. Definition 1.4.1.

A set A is said to be:

1. Finite if A ; Jn for some positive integer n. 2. Countable if A ; Jq. In this case, the set Jq, or any other set equivalent to it, can be used as an index set for A, that is, the elements of A are assigned distinct indices Žsubscripts . that belong to Jq. Hence, A can be represented as A s
a1 , a2 , . . . , a n , . . . 4 .

FINITE, COUNTABLE, AND UNCOUNTABLE SETS

7

3. Uncountable if A is neither finite nor countable. In this case, the I elements of A cannot be indexed by Jn for any n, or by Jq. EXAMPLE 1.4.1. Let A s
1, 4, 9, . . . , n2 , . . . 4 . This set is countable, since the function f : Jq™ A defined by f Ž n. s n2 is one-to-one and onto. Hence, A ; Jq. EXAMPLE 1.4.2. Let A s J be the set of all integers. Then A is countable. To show this, consider the function f : Jq™ A defined by f Ž n. s

½

Ž n q 1 . r2, Ž 2 y n . r2,

n odd, n even.

It can be verified that f is one-to-one and onto. Hence, A ; Jq. EXAMPLE 1.4.3. Let A s
x < 0 F xF 14 . This set is uncountable. To show this, suppose that there exists a one-to-one correspondence between Jq and A. We can then write A s
a1 , a2 , . . . , a n , . . . 4 . Let the digit in the nth decimal place of a n be denoted by bn Ž n s 1, 2, . . . .. Define a number c as c s 0 ⭈ c1 c 2 ⭈⭈⭈ c n ⭈⭈⭈ such that for each n, c n s 1 if bn / 1 and c n s 2 if bn s 1. Now, c belongs to A, since 0 F c F 1. However, by construction, c is different from every a i in at least one decimal digit Ž i s 1, 2, . . . . and hence c f A, which is a contradiction. Therefore, A is not countable. Since A is not finite either, then it must be uncountable. This result implies that any subset of R, the set of real numbers, that contains A, or is equivalent to it, must be uncountable. In particular, R is uncountable. Theorem 1.4.1.

Every infinite subset of a countable set is countable.

Proof. Let A be a countable set, and B be an infinite subset of A. Then A s
a1 , a2 , . . . , a n , . . . 4 , where the a i ’s are distinct elements. Let n1 be the smallest positive integer such that a n1 g B. Let n 2 ) n1 be the next smallest integer such that a n 2 g B. In general, if n1 - n 2 - ⭈⭈⭈ - n ky1 have been chosen, let n k be the smallest integer greater than n ky1 such that a n k g B. Define the function f : Jq™ B such that f Ž k . s a n k, k s 1, 2, . . . . This function is one-to-one and onto. Hence, B is countable. I Theorem 1.4.2.

The union of two countable sets is countable.

Proof. Let A and B be countable sets. Then they can be represented as A s
a1 , a2 , . . . , a n , . . . 4 , B s
b1 , b 2 , . . . , bn , . . . 4 . Define C s A j B. Consider the following two cases: i. A and B are disjoint. ii. A and B are not disjoint.

8

AN INTRODUCTION TO SET THEORY

In case i, let us write C as C s
a1 , b1 , a2 , b 2 , . . . , a n , bn , . . . 4 . Consider the function f : Jq™ C such that f Ž n. s

½

aŽ nq1. r2 ,

n odd,

bn r2 ,

n even.

It can be verified that f is one-to-one and onto. Hence, C is countable. Let us now consider case ii. If A l B / ⭋, then some elements of C, namely those in A l B, will appear twice. Hence, there exists a set E ; Jq such that E ; C. Thus C is either finite or countable. Since C > A and A is infinite, C must be countable. I Corollary 1.4.1. is countable.

If A1 , A 2 , . . . , A n , . . . , are countable sets, then D ⬁is1 A i

Proof. The proof is left as an exercise.

I

Theorem 1.4.3. Let A and B be two countable sets. Then their Cartesian product A = B is countable. Proof. Let us write A as A s Ž a1 , a2 , . . . , a n , . . . 4 . For a given ag A, define Ž a, B . as the set

Ž a, B . s
Ž a, b . < bg B 4 . Then Ž a, B . ; B and hence Ž a, B . is countable. However, ⬁

A = B s D Ž ai , B . . is1

Thus by Corollary 1.4.1, A = B is countable.

I

Corollary 1.4.2. If A1 , A 2 , . . . , A n are countable sets, then their Carten sian product =is1 A i is countable. Proof. The proof is left as an exercise. Corollary 1.4.3.

I

The set Q of all rational numbers is countable.

Proof. By definition, a rational number is a number of the form mrn, ˜ where where m and n are integers with n / 0. Thus Q; Q,

˜
Ž m, n . < m, n are integers and n / 0 4 . Qs

9

BOUNDED SETS

˜ is an infinite subset of J = J, where J is the set of all integers, which Since Q is countable as was seen in Example 1.4.2, then by Theorems 1.4.1 and 1.4.3, ˜ is countable and so is Q. Q I REMARK 1.4.1. Any real number that cannot be expressed as a rational number is called an irrational number. For example, '2 is an irrational number. To show this, suppose that there exist integers, m and n, such that '2 s mrn. We may consider that mrn is written in its lowest terms, that is, m and n have no common factors other than unity. In particular, m and n, cannot both be even. Now, m2 s 2 n2 . This implies that m2 is even. Hence, m is even and can therefore be written as m s 2 m⬘. It follows that n2 s m2r2 s 2 m⬘ 2 . Consequently, n2 , and hence n, is even. This contradicts the fact that m and n are not both even. Thus '2 must be an irrational number.

1.5. BOUNDED SETS Let us consider the set R of real numbers. Definition 1.5.1.

A set A ; R is said to be:

1. Bounded from above if there exists a number q such that xF q for all x in A. This number is called an upper bound of A. 2. Bounded from below if there exists a number p such that xG p for all x in A. The number p is called a lower bound of A. 3. Bounded if A has an upper bound q and a lower bound p. In this case, there exists a nonnegative number r such that yr F xF r for all x in A. This number is equal to maxŽ< p < , < q < .. I Definition 1.5.2. Let A ; R be a set bounded from above. If there exists a number l that is an upper bound of A and is less than or equal to any other upper bound of A, then l is called the least upper bound of A and is denoted by lubŽ A.. Another name for lubŽ A. is the supremum of A and is denoted by supŽ A.. I Definition 1.5.3. Let A ; R be a set bounded from below. If there exists a number g that is a lower bound of A and is greater than or equal to any other lower bound of A, then g is called the greatest lower bound and is denoted by glbŽ A.. The infimum of A, denoted by infŽ A., is another name for glbŽ A.. I The least upper bound of A, if it exists, is unique, but it may or may not belong to A. The same is true for glbŽ A.. The proof of the following theorem is omitted and can be found in Rudin Ž1964, Theorem 1.36..

10

AN INTRODUCTION TO SET THEORY

Theorem 1.5.1.

Let A ; R be a nonempty set.

1. If A is bounded from above, then lubŽ A. exists. 2. If A is bounded from below, then glbŽ A. exists. EXAMPLE 1.5.1. belong to A.

Let A s
x < x- 04 . Then lubŽ A. s 0, which does not

EXAMPLE 1.5.2. Let A s
1rn < n s 1, 2, . . . 4 . Then lubŽ A. s 1 and glbŽ A. s 0. In this case, lubŽ A. belongs to A, but glbŽ A. does not.

1.6. SOME BASIC TOPOLOGICAL CONCEPTS The field of topology is an abstract study that evolved as an independent discipline in response to certain problems in classical analysis and geometry. It provides a unifying theory that can be used in many diverse branches of mathematics. In this section, we present a brief account of some basic definitions and results in the so-called point-set topology. Definition 1.6.1. Let A be a set, and let F s
B␣ 4 be a family of subsets of A. Then F is a topology in A if it satisfies the following properties: 1. The union of any number of members of F is also a member of F. 2. The intersection of a finite number of members of F is also a member of F. 3. Both A and the empty set ⭋ are members of F. I Definition 1.6.2. Let F be a topology in a set A. Then the pair Ž A, F . is called a topological space. I Definition 1.6.3. Let Ž A, F . be a topological space. Then the members of F are called the open sets of the topology F. I Definition 1.6.4. Let Ž A, F . be a topological space. A neighborhood of a point pg A is any open set Žthat is, a member of F . that contains p. In particular, if A s R, the set of real numbers, then a neighborhood of pg R is an open set of the form Nr Ž p . s
q < < q y p < - r 4 for some r ) 0. I Definition 1.6.5. Let Ž A, F . be a topological space. A family G s
B␣ 4 ; F is called a basis for F if each open set Žthat is, member of F . is the union of members of G. I On the basis of this definition, it is easy to prove the following theorem.

11

SOME BASIC TOPOLOGICAL CONCEPTS

Theorem 1.6.1. Let Ž A, F . be a topological space, and let G be a basis for F. Then a set B ; A is open Žthat is, a member of F . if and only if for each pg B, there is a U g G such that pg U ; B. For example, if A s R, then G s
Nr Ž p .< pg R, r ) 04 is a basis for the topology in R. It follows that a set B ; R is open if for every point p in B, there exists a neighborhood Nr Ž p . such that Nr Ž p . ; B. Definition 1.6.6. Let Ž A, F . be a topological space. A set B ; A is closed if B, the complement of B with respect to A, is an open set. I It is easy to show that closed sets of a topological space Ž A, F . satisfy the following properties: 1. The intersection of any number of closed sets is closed. 2. The union of a finite number of closed sets is closed. 3. Both A and the empty set ⭋ are closed. Definition 1.6.7. Let Ž A, F . be a topological space. A point pg A is said to be a limit point of a set B ; A if every neighborhood of p contains at least one element of B distinct from p. Thus, if U Ž p . is any neighborhood of p, then U Ž p . l B is a nonempty set that contains at least one element besides p. In particular, if A s R, the set of real numbers, then p is a limit point of a set B ; R if for any r ) 0, Nr Ž p . l w B y
p4x / ⭋, where
p4 denotes a set consisting of just p. I Theorem 1.6.2. Let p be a limit point of a set B ; R. Then every neighborhood of p contains infinitely many points of B. Proof. The proof is left to the reader.

I

The next theorem is a fundamental theorem in set theory. It is originally due to Bernhard Bolzano Ž1781᎐1848., though its importance was first recognized by Karl Weierstrass Ž1815᎐1897.. The proof is omitted and can be found, for example, in Zaring Ž1967, Theorem 4.62.. Theorem 1.6.3 ŽBolzano᎐Weierstrass .. Every bounded infinite subset of R, the set of real numbers, has at least one limit point. Note that a limit point of a set B may not belong to B. For example, the set B s
1rn < n s 1, 2, . . . 4 has a limit point equal to zero, which does not belong to B. It can be seen here that any neighborhood of 0 contains infinitely many points of B. In particular, if r is a given positive number, then all elements of B of the form 1rn, where n ) 1rr, belong to Nr Ž0.. From Theorem 1.6.2 it can also be concluded that a finite set cannot have limit points.

12

AN INTRODUCTION TO SET THEORY

Limit points can be used to describe closed sets, as can be seen from the following theorem. Theorem 1.6.4. belongs to B.

A set B is closed if and only if every limit point of B

Proof. Suppose that B is closed. Let p be a limit point of B. If pf B, then pg B, which is open. Hence, there exists a neighborhood U Ž p . of p contained inside B by Theorem 1.6.1. This means that U Ž p . l B s ⭋, a contradiction, since p is a limit point of B Žsee Definition 1.6.7.. Therefore, p must belong to B. Vice versa, if every limit point of B is in B, then B must be closed. To show this, let p be any point in B. Then, p is not a limit point of B. Therefore, there exists a neighborhood U Ž p . such that U Ž p . ; B. This means that B is open and hence B is closed. I It should be noted that a set does not have to be either open or closed; if it is closed, it does not have to be open, and vice versa. Also, a set may be both open and closed. EXAMPLE 1.6.1. B s
x < 0 - x- 14 is an open subset of R, but is not closed, since both 0 and 1 are limit points of B, but do not belong to it. EXAMPLE 1.6.2. B s
x < 0 F xF 14 is closed, but is not open, since any neighborhood of 0 or 1 is not contained in B. EXAMPLE 1.6.3. B s
x < 0 - xF 14 is not open, because any neighborhood of 1 is not contained in B. It is also not closed, because 0 is a limit point that does not belong to B. EXAMPLE 1.6.4.

The set R is both open and closed.

EXAMPLE 1.6.5. A finite set is closed because it has no limit points, but is obviously not open. Definition 1.6.8. A subset B of a topological space Ž A, F . is disconnected if there exist open subsets C and D of A such that B l C and B l D are disjoint nonempty sets whose union is B. A set is connected if it is not disconnected. I The set of all rationals Q is disconnected, since
x < x ) '2 4 l Q and
x < x - '2 4 l Q are disjoint nonempty sets whose union is Q. On the other hand, all intervals in R Žopen, closed, or half-open. are connected. Definition 1.6.9. A collection of sets
B␣ 4 is said to be a co®ering of a set A if the union D␣ B␣ contains A. If each B␣ is an open set, then
B␣ 4 is called an open co®ering.

13

EXAMPLES IN PROBABILITY AND STATISTICS

Definition 1.6.10. A set A in a topological space is compact if each open covering
B␣ 4 of A has a finite subcovering, that is, there is a finite n subcollection B␣ 1, B␣ 2 , . . . , B␣ n of
B␣ 4 such that A ; D is1 B␣ i . I The concept of compactness is motivated by the classical Heine᎐Borel theorem, which characterizes compact sets in R, the set of real numbers, as closed and bounded sets. Theorem 1.6.5 ŽHeine᎐Borel.. closed and bounded.

A set B ; R is compact if and only if it is

Proof. See, for example, Zaring Ž1967, Theorem 4.78..

I

Thus, according to the Heine᎐Borel theorem, every closed and bounded interval w a, b x is compact.

1.7. EXAMPLES IN PROBABILITY AND STATISTICS EXAMPLE 1.7.1. In probability theory, events are considered as subsets in a sample space ⍀, which consists of all the possible outcomes of an experiment. A Borel field of events Žalso called a ␴-field. in ⍀ is a collection B of events with the following properties: i. ⍀ g B. ii. If E g B, then E g B, where E is the complement of E. iii. If E1 , E2 , . . . , En , . . . is a countable collection of events in B, then D ⬁is1 Ei belongs to B. The probability of an event E is a number denoted by P Ž E . that has the following properties: i. 0 F P Ž E . F 1. ii. P Ž ⍀ . s 1. iii. If E1 , E2 , . . . , En , . . . is a countable collection of disjoint events in B, then

ž / ⬁

P

D Ei is1

s

⬁

Ý P Ž Ei . .

is1

By definition, the triple Ž ⍀, B, P . is called a probability space. EXAMPLE 1.7.2 . A random variable X defined on a probability space Ž ⍀, B, P . is a function X: ⍀ ™ A, where A is a nonempty set of real numbers. For any real number x, the set E s
␻ g ⍀ < X Ž ␻ . F x 4 is an

14

AN INTRODUCTION TO SET THEORY

element of B. The probability of the event E is called the cumulative distribution function of X and is denoted by F Ž x .. In statistics, it is customary to write just X instead of X Ž ␻ .. We thus have FŽ x. sPŽ XFx. . This concept can be extended to several random variables: Let X 1 , X 2 , . . . , X n be n random variables. Define the event A i s
␻ g ⍀ < X i Ž ␻ . F x i 4 , i s n 1, 2, . . . , n. Then, P ŽF is1 A i ., which can be expressed as F Ž x 1 , x 2 , . . . , x n . s P Ž X1 F x 1 , X 2 F x 2 , . . . , X n F x n . , is called the joint cumulative distribution function of X 1 , X 2 , . . . , X n . In this case, the n-tuple Ž X 1 , X 2 , . . . , X n . is said to have a multivariate distribution. A random variable X is said to be discrete, or to have a discrete distribution, if its range is finite or countable. For example, the binomial random variable is discrete. It represents the number of successes in a sequence of n independent trials, in each of which there are two possible outcomes: success or failure. The probability of success, denoted by pn , is the same in all the trials. Such a sequence of trials is called a Bernoulli sequence. Thus the possible values of this random variable are 0, 1, . . . , n. Another example of a discrete random variable is the Poisson, whose possible values are 0, 1, 2, . . . . It is considered to be the limit of a binomial random variable as n ™ ⬁ in such a way that npn ™ ␭ ) 0. Other examples of discrete random variables include the discrete uniform, geometric, hypergeometric, and negative binomial Žsee, for example, Fisz, 1963; Johnson and Kotz, 1969; Lindgren 1976; Lloyd, 1980.. A random variable X is said to be continuous, or to have a continuous distribution, if its range is an uncountable set, for example, an interval. In this case, the cumulative distribution function F Ž x . of X is a continuous function of x on the set R of all real numbers. If, in addition, F Ž x . is differentiable, then its derivative is called the density function of X. One of the best-known continuous distributions is the normal. A number of continuous distributions are derived in connection with it, for example, the chisquared, F, Rayleigh, and t distributions. Other well-known continuous distributions include the beta, continuous uniform, exponential, and gamma distributions Žsee, for example, Fisz, 1963; Johnson and Kotz, 1970a, b.. EXAMPLE 1.7.3. Let f Ž x, ␪ . denote the density function of a continuous random variable X, where ␪ represents a set of unknown parameters that identify the distribution of X. The range of X, which consists of all possible values of X, is referred to as a population and denoted by PX . Any subset of n elements from PX forms a sample of size n. This sample is actually an element in the Cartesian product PXn . Any real-valued function defined on PXn is called a statistic. We denote such a function by g Ž X 1 , X 2 , . . . , X n ., where each X i has the same distribution as X. Note that this function is a random variable whose values do not depend on ␪ . For example, the sample mean X s Ý nis1 X irn and the sample variance S 2 s Ý nis1 Ž X i y X . 2rŽ n y 1.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

15

are statistics. We adopt the convention that whenever a particular sample of size n is chosen Žor observed. from PX , the elements in that sample are written using lowercase letters, for example, x 1 , x 2 , . . . , x n . The corresponding value of a statistic is written as g Ž x 1 , x 2 , . . . , x n .. EXAMPLE 1.7.4. Two random variables, X and Y, are said to be equal in distribution if they have the same cumulative distribution function. This fact d is denoted by writing X s Y. The same definition applies to random variables d with multivariate distributions. We note that s is an equivalence relation, since it satisfies properties 1, 2, and 3 in Definition 1.3.2. The first two d d properties are obviously true. As for property 3, if X s Y and Y s Z, then d X s Z, which implies that all three random variables have the same cumulative distribution function. This equivalence relation is useful in nonparametric statistics Žsee Randles and Wolfe, 1979.. For example, it can be shown that if X has a distribution that is symmetric about some number ␮ , then d X y ␮ s ␮ y X. Also, if X 1 , X 2 , . . . , X n are independent and identically distributed random variables, and if Ž m1 , m 2 , . . . , m n . is any permutation of the d n-tuple Ž1, 2, . . . , n., then Ž X 1 , X 2 , . . . , X n . s Ž X m 1, X m 2 , . . . , X m n .. In this case, we say that the collection of random variables X 1 , X 2 , . . . , X n is exchangeable. EXAMPLE 1.7.5. Consider the problem of testing the null hypothesis H0 : ␪ F ␪ 0 versus the alternative hypothesis Ha : ␪ ) ␪ 0 , where ␪ is some unknown parameter that belongs to a set A. Let T be a statistic used in making a decision as to whether H0 should be rejected or not. This statistic is appropriately called a test statistic. Suppose that H0 is rejected if T ) t, where t is some real number. Since the distribution of T depends on ␪ , then the probability P ŽT ) t . is a function of ␪ , which we denote by ␲ Ž ␪ .. Thus ␲ : A ™ w0,1x. Let B0 be a subset of A defined as B0 s
␪ g A < ␪ F ␪ 0 4 . By definition, the size of the test is the least upper bound of the set ␲ Ž B0 .. This probability is denoted by ␣ and is also called the level of significance of the test. We thus have

␣ s sup ␲ Ž ␪ . . ␪F␪ 0

To learn more about the above examples and others, the interested reader may consider consulting some of the references listed in the annotated bibliography.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

Bronshtein, I. N., and K. A. Semendyayev Ž1985.. Handbook of Mathematics ŽEnglish translation edited by K. A. Hirsch.. Van Nostrand Reinhold, New York. ŽSection 4.1 in this book gives basic concepts of set theory; Chap. 5 provides a brief introduction to probability and mathematical statistics. .

16

AN INTRODUCTION TO SET THEORY

Dugundji, J. Ž1966.. Topology. Allyn and Bacon, Boston. ŽChap. 1 deals with elementary set theory; Chap. 3 presents some basic topological concepts that complements the material given in Section 1.6.. Fisz, M. Ž1963.. Probability Theory and Mathematical Statistics, 3rd ed. Wiley, New York. ŽChap. 1 discusses random events and axioms of the theory of probability; Chap. 2 introduces the concept of a random variable; Chap. 5 investigates some probability distributions. . Hardy, G. H. Ž1955.. A Course of Pure Mathematics, 10th ed. The University Press, Cambridge, England. ŽChap. 1 in this classic book is recommended reading for understanding the real number system.. Harris, B. Ž1966.. Theory of Probability. Addison-Wesley, Reading, Massachusetts. ŽChaps. 2 and 3 discuss some elementary concepts in probability theory as well as in distribution theory. Many exercises are provided.. Hogg, R. V., and A. T. Craig Ž1965.. Introduction to Mathematical Statistics, 2nd ed. Macmillan, New York. ŽChap. 1 is an introduction to distribution theory; examples of some special distributions are given in Chap. 3; Chap. 10 considers some aspects of hypothesis testing that pertain to Example 1.7.5.. Johnson, N. L., and S. Kotz Ž1969.. Discrete Distributions. Houghton Mifflin, Boston. ŽThis is the first volume in a series of books on statistical distributions. It is an excellent source for getting detailed accounts of the properties and uses of these distributions. This volume deals with discrete distributions, including the binomial in Chap. 3, the Poisson in Chap. 4, the negative binomial in Chap. 5, and the hypergeometric in Chap. 6.. Johnson, N. L., and S. Kotz Ž1970a.. Continuous Uni®ariate Distributionsᎏ1. Houghton Mifflin, Boston. ŽThis volume covers continuous distributions, including the normal in Chap. 13, lognormal in Chap. 14, Cauchy in Chap. 16, gamma in Chap. 17, and the exponential in Chap. 18.. Johnson, N. L., and S. Kotz Ž1970b.. Continuous Uni®ariate Distributionsᎏ2. Houghton Mifflin, Boston. ŽThis is a continuation of Vol. 2 on continuous distributions. Chaps. 24, 25, 26, and 27 discuss the beta, continuous uniforms, F, and t distributions, respectively. . Johnson, P. E. Ž1972.. A History of Set Theory. Prindle, Weber, and Schmidt, Boston. ŽThis book presents a historical account of set theory as was developed by Georg Cantor.. Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽSections 1.1, 1.2, 2.1, 3.1, 3.2, and 3.3 present introductory material on probability models and distributions; Chap. 6 discusses test of hypothesis and statistical inference. . Lloyd, E. Ž1980.. Handbook of Applicable Mathematics, Vol. II. Wiley, New York. ŽThis is the second volume in a series of six volumes designed as texts of mathematics for professionals. Chaps. 1, 2, and 3 present expository material on probability; Chaps. 4 and 5 discuss random variables and their distributions. . Randles, R. H., and D. A. Wolfe Ž1979.. Introduction to the Theory of Nonparametric Statistics. Wiley, New York. ŽSection 1.3 in this book discusses the ‘‘equal in distribution’’ property mentioned in Example 1.7.4.. Rudin, W. Ž1964.. Principles of Mathematical Analysis, 2nd ed. McGraw-Hill, New York. ŽChap. 1 discusses the real number system; Chap. 2 deals with countable, uncountable, and bounded sets and pertains to Sections 1.4, 1.5, and 1.6..

EXERCISES

17

Stoll, R. R. Ž1963.. Set Theory and Logic. W. H. Freeman, San Francisco. ŽChap. 1 is an introduction to set theory; Chap. 2 discusses countable sets; Chap. 3 is useful in understanding the real number system.. Tucker, H. G. Ž1962.. Probability and Mathematical Statistics. Academic Press, New York. ŽChaps. 1, 3, 4, and 6 discuss basic concepts in elementary probability and distribution theory.. Vilenkin, N. Y. Ž1968.. Stories about Sets. Academic Press, New York. ŽThis is an interesting book that presents various notions of set theory in an informal and delightful way. It contains many unusual stories and examples that make the learning of set theory rather enjoyable.. Zaring, W. M. Ž1967.. An Introduction to Analysis. Macmillan, New York. ŽChap. 2 gives an introduction to set theory; Chap. 3 discusses functions and relations. .

EXERCISES In Mathematics 1.1. Verify Results 1.2.3᎐1.2.12. 1.2. Verify Results 1.2.13᎐1.2.16. 1.3. Let A, B, and C be sets such that A l B ; C and A j C ; B. Show that A and C are disjoint. 1.4. Let A, B, and C be sets such that C s Ž A y B . j Ž B y A.. The set C is called the symmetric difference of A and B and is denoted by A ` B. Show that (a) A^ B s A j B y A l B (b) A^Ž B^ D . s Ž A^ B .^ D, where D is any set. (c) A l Ž B^ D . s Ž A l B .^Ž A l D ., where D is any set. 1.5. Let A s Jq= Jq, where Jq is the set of positive integers. Define a relation ␳ in A as follows: If Ž m1 , n1 . and Ž m 2 , n 2 . are elements in A, then Ž m1 , n1 . ␳ Ž m 2 , n 2 . if m1 n 2 s n1 m 2 . Show that ␳ is an equivalence relation and describe its equivalence classes. 1.6. Let A be the same set as in Exercise 1.5. Show that the following relation is an equivalence relation: Ž m1 , n1 . ␳ Ž m 2 , n 2 . if m1 q n 2 s n1 q m 2 . Draw the equivalence class of Ž1, 2.. 1.7. Consider the set A s
Žy2,y 5., Žy1,y 3., Ž1, 2., Ž3, 10.4 . Show that A defines a function. 1.8. Let A and B be two sets and f be a function defined on A such that f Ž A. ; B. If A1 , A 2 , . . . , A n are subsets of A, then show that: n n (a) f ŽD is1 A i . s D is1 f Ž A i ..

18

AN INTRODUCTION TO SET THEORY n n (b) f ŽF is1 A i . ; F is1 f Ž A i .. Under what conditions are the two sides in Žb. equal?

1.9. Prove Corollary 1.4.1. 1.10. Prove Corollary 1.4.2. 1.11. Show that the set A s
3, 9, 19, 33, 51, 73, . . . 4 is countable. 1.12. Show that '3 is an irrational number. 1.13. Let a, b, c, and d be rational numbers such that aq 'b s c q 'd . Then, either (a) as c, bs d, or (b) b and d are both squares of rational numbers. 1.14. Let A ; R be a nonempty set bounded from below. Define yA to be the set
yx < x g A4 . Show that infŽ A. s ysupŽyA.. 1.15. Let A ; R be a closed and bounded set, and let supŽ A. s b. Show that bg A. 1.16. Prove Theorem 1.6.2. 1.17. Let Ž A, F . be a topological space. Show that G ; F is a basis for F in and only if for each B g F and each pg B, there is a U g G such that pg U ; B. 1.18. Show that if A and B are closed sets, then A j B is a closed set. 1.19. Let B ; A be a closed subset of a compact set A. Show that B is compact. 1.20. Is a compact subset of a compact set necessarily closed? In Statistics 1.21. Let X be a random variable. Consider the following events: A n s
␻ g ⍀ < X Ž ␻ . - xq 3yn 4 ,

n s 1, 2, . . . ,

Bn s
␻ g ⍀ < X Ž ␻ . F xy 3

n s 1, 2, . . . ,

yn

A s
␻ g ⍀< X Ž ␻ . F x4 , B s
␻ g ⍀< X Ž ␻ . - x4 ,

4,

19

EXERCISES

where x is a real number. Show that for any x, (a) F ⬁ns1 A n s A; (b) D ⬁ns1 Bn s B. 1.22. Let X be a nonnegative random variable such that E Ž X . s ␮ is finite, where E Ž X . denotes the expected value of X. The following inequality, known as Marko®’s inequality, is true: ␮ P Ž X G h. F , h where h is any positive number. Consider now a Poisson random variable with parameter ␭. (a) Find an upper bound on the probability P Ž X G 2. using Markov’s inequality. (b) Obtain the exact probability value in Ža., and demonstrate that it is smaller than the corresponding upper bound in Markov’s inequality. 1.23. Let X be a random variable whose expected value ␮ and variance ␴ 2 exist. Show that for any positive constants c and k, (a) P Ž< X y ␮ < G c . F ␴ 2rc 2 , (b) P Ž< X y ␮ < G k ␴ . F 1rk 2 , (c) P Ž< X y ␮ < - k ␴ . G 1 y 1rk 2 . The preceding three inequalities are equivalent versions of the so-called Chebyshe®’s inequality. 1.24. Let X be a continuous random variable with the density function f Ž x. s

½

1y < x< , 0

y1 - x- 1, elsewhere .

By definition, the density function of X is a nonnegative function such x that F Ž x . s Hy⬁ f Ž t . dt, where F Ž x . is the cumulative distribution function of X. (a) Apply Markov’s inequality to finding upper bounds on the following probabilities: Ži. P Ž< X < G 12 .; Žii. P Ž< X < ) 13 .. (b) Compute the exact value of P Ž< X < G 21 ., and compare it against the upper bound in Ža.Ži.. 1.25. Let X 1 , X 2 , . . . , X n be n continuous random variables. Define the random variables XŽ1. and XŽ n. as XŽ1. s min
X 1 , X 2 , . . . , X n 4 , 1FiFn

XŽ n. s max
X 1 , X 2 , . . . , X n 4 . 1FiFn

20

AN INTRODUCTION TO SET THEORY

Show that for any x, (a) P Ž XŽ1. G x . s P Ž X 1 G x, X 2 G x, . . . , X n G x ., (b) P Ž XŽ n. F x . s P Ž X 1 F x, X 2 F x, . . . , X n F x .. In particular, if X 1 , X 2 , . . . , X n form a sample of size n from a population with a cumulative distribution function F Ž x ., show that (c) P Ž XŽ1. F x . s 1 y w1 y F Ž x .x n, (d) P Ž XŽ n. F x . s w F Ž x .x n. The statistics XŽ1. and XŽ n. are called the first-order and nth-order statistics, respectively. 1.26. Suppose that we have a sample of size n s 5 from a population with an exponential distribution whose density function is

½

y2 x , f Ž x. s 2e 0

Find the value of P Ž2 F XŽ1. F 3..

x) 0, elsewhere .

CHAPTER 2

Basic Concepts in Linear Algebra

In this chapter we present some fundamental concepts concerning vector spaces and matrix algebra. The purpose of the chapter is to familiarize the reader with these concepts, since they are essential to the understanding of some of the remaining chapters. For this reason, most of the theorems in this chapter will be stated without proofs. There are several excellent books on linear algebra that can be used for a more detailed study of this subject Žsee the bibliography at the end of this chapter .. In statistics, matrix algebra is used quite extensively, especially in linear models and multivariate analysis. The books by Basilevsky Ž1983., Graybill Ž1983., Magnus and Neudecker Ž1988., and Searle Ž1982. include many applications of matrices in these areas. In this chapter, as well as in the remainder of the book, elements of the set of real numbers, R, are sometimes referred to as scalars. The Cartesian n product =is1 R is denoted by R n, which is also known as the n-dimensional Euclidean space. Unless otherwise stated, all matrix elements are considered to be real numbers.

2.1. VECTOR SPACES AND SUBSPACES A vector space over R is a set V of elements called vectors together with two operations, addition and scalar multiplication, that satisfy the following conditions: 1. 2. 3. 4. 5.

u q v is an element of V for all u, v in V. If ␣ is a scalar and u g V, then ␣ u g V. u q v s v q u for all u, v in V. u q Žv q w. s Žu q v. q w for all u, v, w in V. There exists an element 0 g V such that 0 q u s u for all u in V. This element is called the zero vector. 21

22

BASIC CONCEPTS IN LINEAR ALGEBRA

6. 7. 8. 9. 10.

For each u g V there exists a v g V such that u q v s 0. ␣ Žu q v. s ␣ u q ␣ v for any scalar ␣ and any u and v in V. Ž ␣ q ␤ .u s ␣ u q ␤ u for any scalars ␣ and ␤ and any u in V. ␣ Ž ␤ u. s Ž ␣␤ .u for any scalars ␣ and ␤ and any u in V. 1u s u for any u g V.

EXAMPLE 2.1.1. A familiar example of a vector space is the n-dimensional Euclidean space R n. Here, addition and multiplication are defined as follows: If Ž u1 , u 2 , . . . , u n . and Ž ®1 , ®2 , . . . , ®n . are two elements in R n, then their sum is defined as Ž u1 q ®1 , u 2 q ®2 , . . . , u n q ®n .. If ␣ is a scalar, then ␣ Ž u 1 , u 2 , . . . , u n . s Ž ␣ u 1 , ␣ u 2 , . . . , ␣ u n .. EXAMPLE 2.1.2. Let V be the set of all polynomials in x of degree less than or equal to k. Then V is a vector space. Any element in V can be expressed as Ý kis0 a i x i, where the a i ’s are scalars. EXAMPLE 2.1.3. Let V be the set of all functions defined on the closed interval wy1, 1x. Then V is a vector space. It can be seen that f Ž x . q g Ž x . and ␣ f Ž x . belong to V, where f Ž x . and g Ž x . are elements in V and ␣ is any scalar. EXAMPLE 2.1.4. The set V of all nonnegative functions defined on wy1, 1x is not a vector space, since if f Ž x . g V and ␣ is a negative scalar, then ␣ f Ž x . f V. EXAMPLE 2.1.5. Let V be the set of all points Ž x, y . on a straight line given by the equation 2 xy y q 1 s 0. Then V is not a vector space. This is because if Ž x 1 , y 1 . and Ž x 2 , y 2 . belong to V, then Ž x 1 q x 2 , y 1 q y 2 . f V, since 2Ž x 1 q x 2 . y Ž y 1 q y 2 . q 1 s y1 / 0. Alternatively, we can state that V is not a vector space because the zero element Ž0, 0. does not belong to V. This violates condition 5 for a vector space. A subset W of a vector space V is said to form a vector subspace if W itself is a vector space. Equivalently, W is a subspace if whenever u, v g W and ␣ is a scalar, then u q v g W and ␣ u g W. For example, the set W of all continuous functions defined on wy1, 1x is a vector subspace of V in Example 2.1.3. Also, the set of all points on the straight line y y 2 xs 0 is a vector subspace of R 2 . However, the points on any straight line in R 2 not going through the origin Ž0, 0. do not form a vector subspace, as was seen in Example 2.1.5. Definition 2.1.1. Let V be a vector space, and u 1 , u 2 , . . . , u n be a collection of n elements in V. These elements are said to be linearly dependent if there exist n scalars ␣ 1 , ␣ 2 , . . . , ␣ n , not all equal to zero, such that Ý nis1 ␣ i u i s 0. If, however, Ý nis1 ␣ i u i s 0 is true only when all the ␣ i ’s are zero, then

23

VECTOR SPACES AND SUBSPACES

u 1 , u 2 , . . . , u n are linearly independent. It should be noted that if u 1 , u 2 , . . . , u n are linearly independent, then none of them can be zero. If, for example, u 1 s 0, then ␣ u 1 q 0u 2 q ⭈⭈⭈ q0u n s 0 for any ␣ / 0, which implies that the u i ’s are linearly dependent, a contradiction. I From the preceding definition we can say that a collection of n elements in a vector space are linearly dependent if at least one element in this collection can be expressed as a linear combination of the remaining n y 1 elements. If no element, however, can be expressed in this fashion, then the n elements are linearly independent. For example, in R 3, Ž1, 2,y 2., Žy1, 0, 3., and Ž1, 4,y 1. are linearly dependent, since 2Ž1, 2,y 2. q Žy1, 0, 3. y Ž1, 4, y1. s 0. On the other hand, it can be verified that Ž1, 1, 0., Ž1, 0, 2., and Ž0, 1, 3. are linearly independent. Definition 2.1.2. Let u 1 , u 2 , . . . , u n be n elements in a vector space V. The collection of all linear combinations of the form Ý nis1 ␣ i u i , where the ␣ i ’s are scalars, is called a linear span of u 1 , u 2 , . . . , u n and is denoted by LŽu 1 , u 2 , . . . , u n .. I It is easy to see from the preceding definition that LŽu 1 , u 2 , . . . , u n . is a vector subspace of V. This vector subspace is said to be spanned by u1, u 2 , . . . , u n. Definition 2.1.3. Let V be a vector space. If there exist linearly independent elements u 1 , u 2 , . . . , u n in V such that V s LŽu 1 , u 2 , . . . , u n ., then u 1 , u 2 , . . . , u n are said to form a basis for V. The number n of elements in this basis is called the dimension of the vector space and is denoted by dim V. I Note that a basis for a vector space is not unique. However, its dimension is unique. For example, the three vectors Ž1, 0, 0., Ž0, 1, 0., and Ž0, 0, 1. form a basis for R 3. Another basis for R 3 consists of Ž1, 1, 0., Ž1, 0, 1., and Ž0, 1, 1.. If u 1 , u 2 , . . . , u n form a basis for V and if u is a given element in V, then there exists a unique set of scalars, ␣ 1 , ␣ 2 , . . . , ␣ n , such that u s Ý nis1 ␣ i u i . To show this, suppose that there exists another set of scalars, ␤ 1 , ␤ 2 , . . . , ␤n , such that u s Ý nis1 ␤i u. Then Ý nis1 Ž ␣ i y ␤i .u i s 0, which implies that ␣ i s ␤i for all i, since the u i ’s are linearly independent. Let us now check the dimensions of the vector spaces for some of the examples described earlier. For Example 2.1.1, dim V s n. In Example 2.1.2,
1, x, x 2 , . . . , x k 4 is a basis for V; hence dim V s k q 1. As for Example 2.1.3, dim V is infinite, since there is no finite set of functions that can span V. Definition 2.1.4. Let u and v be two vectors in R n. The dot product Žalso called scalar product or inner product. of u and v is a scalar denoted by u ⭈ v and is given by n

u⭈vs

Ý u i ®i ,

is1

24

BASIC CONCEPTS IN LINEAR ALGEBRA

where u i and ®i are the ith components of u and v, respectively Ž i s 1, 2, . . . , n.. In particular, if u s v, then Žu ⭈ u.1r2 s ŽÝ nis1 u 2i .1r2 is called the Euclidean norm Žor length. of u and is denoted by 5 u 5 2 . The dot product of u and v is also equal to 5 u 5 2 5 v 5 2 cos ␪ , where ␪ is the angle between u and v. I Definition 2.1.5. Two vectors u and v in R n are said to be orthogonal if their dot product is zero. I Definition 2.1.6. Let U be a vector subspace of R n. The vectors e 1 , e 2 , . . . , e m form an orthonormal basis for U if they satisfy the following properties: 1. e 1 , e 2 , . . . , e m form a basis for U. 2. e i ⭈ e j s 0 for all i / j Ž i, j s 1, 2, . . . , m.. 3. 5 e i 5 2 s 1 for i s 1, 2, . . . , m. Any collection of vectors satisfying just properties 2 and 3 are said to be orthonormal. I Theorem 2.1.1. Let u 1 , u 2 , . . . , u m be a basis for a vector subspace U of R n. Then there exists an orthonormal basis, e 1 , e 2 , . . . , e m , for U, given by e1 s e2 s

em s

v1 5 v1 5 2 v2 5 v2 5 2 . . . vm 5 vm 5 2

,

where v1 s u 1 ,

,

where v2 s u 2 y

,

where vm s u m y

v1 ⭈ u 2

v1 ,

5 v1 5 22

my1

Ý

vi ⭈ u m

is1

Proof. See Graybill Ž1983, Theorem 2.6.5..

5 vi 5 22

vi .

I

The procedure of constructing an orthonormal basis from any given basis as described in Theorem 2.1.1 is known as the Gram-Schmidt orthonormalization procedure. Theorem 2.1.2.

Let u and v be two vectors in R n. Then:

1. < u ⭈ v < F 5 u 5 2 5 v 5 2 . 2. 5 u q v 5 2 F 5 u 5 2 q 5 v 5 2 . Proof. See Marcus and Minc Ž1988, Theorem 3.4..

I

25

LINEAR TRANSFORMATIONS

The inequality in part 1 of Theorem 2.1.2 is known as the Cauchy᎐Schwarz inequality. The one in part 2 is called the triangle inequality. Definition 2.1.7. Let U be a vector subspace of R n. The orthogonal complement of U, denoted by U H, is the vector subspace of R n which consists of all vectors v such that u ⭈ v s 0 for all u in U. I Definition 2.1.8. Let U1 , U2 , . . . , Un be vector subspaces of the vector n space U. The direct sum of these vector subspaces, denoted by [is1 Ui , n consists of all vectors u that can be uniquely expressed as u s Ý is1 u i , where u i g Ui , i s 1, 2, . . . , n. I Theorem 2.1.3. U. Then:

Let U1 , U2 , . . . , Un be vector subspaces of the vector space

n 1. [is1 Ui is a vector subspace of U. n n 2. If U s [is1 Ui , then F is1 Ui consists of just the zero element 0 of U. n n 3. dim [is1 Ui s Ý is1 dim Ui .

Proof. The proof is left as an exercise. Theorem 2.1.4.

I

Let U be a vector subspace of R n. Then R n s U [ U H .

Proof. See Marcus and Minc Ž1988, Theorem 3.3..

I

From Theorem 2.1.4 we conclude that any v g R n can be uniquely written as v s v1 q v2 , where v1 g U and v2 g U H . In this case, v1 and v2 are called the projections of v on U and U H , respectively. 2.2. LINEAR TRANSFORMATIONS Let U and V be two vector spaces. A function T : U ™ V is called a linear transformation if T Ž ␣ 1u 1 q ␣ 2 u 2 . s ␣ 1T Žu 1 . q ␣ 2 T Žu 2 . for all u 1 , u 2 in U and any scalars ␣ 1 and ␣ 2 . For example, let T : R 3 ™ R 3 be defined as T Ž x1 , x 2 , x 3 . s Ž x1 y x 2 , x1 q x 3 , x 3 . . Then T is a linear transformation, since T ␣ Ž x1 , x 2 , x 3 . q ␤ Ž y1 , y 2 , y 3 . s T Ž ␣ x1 q ␤ y1 , ␣ x 2 q ␤ y 2 , ␣ x 3 q ␤ y3 . s Ž ␣ x1 q ␤ y1 y ␣ x 2 y ␤ y 2 , ␣ x1 q ␤ y1 q ␣ x 3 q ␤ y 3 , ␣ x 3 q ␤ y 3 . s ␣ Ž x1 y x 2 , x1 q x 3 , x 3 . q ␤ Ž y1 y y 2 , y1 q y 3 , y 3 . s ␣ T Ž x1 , x 2 , x 3 . q ␤ T Ž y1 , y 2 , y 3 . .

26

BASIC CONCEPTS IN LINEAR ALGEBRA

We note that the image of U under T, or the range of T, namely T ŽU ., is a vector subspace of V. This is true because if v1 , v2 are in T ŽU ., then there exist u 1 and u 2 in U such that v1 s T Žu 1 . and v2 s T Žu 2 .. Hence, v1 q v2 s T Žu 1 . q T Žu 2 . s T Žu 1 q u 2 ., which belongs to T ŽU .. Also, if ␣ is a scalar, then ␣ T Žu. s T Ž ␣ u. g T ŽU . for any u g U. Definition 2.2.1. Let T : U ™ V be a linear transformation. The kernel of T, denoted by ker T, is the collection of all vectors u in U such that T Žu. s 0, where 0 is the zero vector in V. The kernel of T is also called the null space of T. As an example of a kernel, let T : R 3 ™ R 3 be defined as T Ž x 1 , x 2 , x 3 . s Ž x 1 y x 2 , x 1 y x 3 .. Then ker T s
Ž x 1 , x 2 , x 3 . < x 1 s x 2 , x 1 s x 3 4 In this case, ker T consists of all points Ž x 1 , x 2 , x 3 . in R 3 that lie on a straight line through the origin given by the equations x 1 s x 2 s x 3 . I Let T : U ™ V be a linear transformation. Then we have

Theorem 2.2.1. the following:

1. ker T is a vector subspace of U. 2. dim U s dimŽker T . q dimw T ŽU .x. Proof. Part 1 is left as an exercise. To prove part 2 we consider the following. Let dim U s n, dimŽker T . s p, and dimw T ŽU .x s q. Let u 1 , u 2 , . . . , u p be a basis for ker T, and v1 , v2 , . . . , vq be a basis for T ŽU .. Then, there exist vectors w 1 , w 2 , . . . , wq in U such that T Žwi . s vi Ž i s 1, 2, . . . , q .. We need to show that u 1 , u 2 , . . . , u p ; w 1 , w 2 , . . . , wq form a basis for U, that is, they are linearly independent and span U. Suppose that there exist scalars ␣ 1 , ␣ 2 , . . . , ␣ p ; ␤ 1 , ␤ 2 , . . . , ␤ q such that p

q

is1

is1

Ý ␣ i u i q Ý ␤i wi s 0.

Then 0sT

ž

p

q

is1

is1

Ý ␣ i u i q Ý ␤ i wi

/

Ž 2.1 .

,

where 0 represents the zero vector in V s s

p

q

is1 q

is1

Ý ␣ i T Ž u i . q Ý ␤ i T Ž wi . Ý ␤ i T Ž wi . ,

is1 q

s

Ý ␤i vi .

is1

since

u i g ker T , i s 1, 2, . . . , p

27

MATRICES AND DETERMINANTS

Since the vi ’s are linearly independent, then ␤i s 0 for i s 1, 2, . . . , q. From Ž2.1. it follows that ␣ i s 0 for i s 1, 2, . . . , p, since the u i ’s are also linearly independent. Thus the vectors u 1 , u 2 , . . . , u p ; w 1 , w 2 , . . . , wq are linearly independent. Let us now suppose that u is any vector in U. To show that it belongs to LŽu 1 , u 2 , . . . , u p ; w 1 , w 2 , . . . , wq .. Let v s T Žu.. Then there exist scalars a1 , a2 , . . . , a q such that v s Ý qis1 a i vi . It follows that q

T Ž u. s

Ý a i T Ž wi .

is1

žÝ / q

sT

a i wi .

is1

Thus,

ž

q

T uy

Ý a i wi

is1

/

s 0,

and u y Ý qis1 a i wi must then belong to ker T. Hence, uy

q

p

is1

is1

Ý ai wi s Ý bi u i

Ž 2.2 .

for some scalars, b1 , b 2 , . . . , bp . From Ž2.2. we then have us

p

q

is1

is1

Ý bi u i q Ý ai wi ,

which shows that u belongs to the linear span of u 1 , u 2 , . . . , u p ; w 1 , w 2 , . . . , wq . We conclude that these vectors form a basis for U. Hence, n s pq q. I Corollary 2.2.1. T : U ™ V is a one-to-one linear transformation if and only if dimŽker T . s 0. Proof. If T is a one-to-one linear transformation, then ker T consists of just one vector, namely, the zero vector. Hence, dimŽker T . s 0. Vice versa, if dimŽker T . s 0, or equivalently, if ker T consists of just the zero vector, then T must be a one-to-one transformation. This is true because if u 1 and u 2 are in U and such that T Žu 1 . s T Žu 2 ., then T Žu 1 y u 2 . s 0, which implies that u 1 y u 2 g ker T and thus u 1 y u 2 s 0. I 2.3. MATRICES AND DETERMINANTS Matrix algebra was devised by the English mathematician Arthur Cayley Ž1821᎐1895.. The use of matrices originated with Cayley in connection with

28

BASIC CONCEPTS IN LINEAR ALGEBRA

linear transformations of the form ax 1 q bx 2 s y 1 , cx 1 q dx 2 s y 2 , where a, b, c, and d are scalars. This transformation is completely determined by the square array a c

b , d

which is called a matrix of order 2 = 2. In general, let T : U ™ V be a linear transformation, where U and V are vector spaces of dimensions m and n, respectively. Let u 1 , u 2 , . . . , u m be a basis for U and v1 , v2 , . . . , vn be a basis for V. For i s 1, 2, . . . , m, consider T Žu i ., which can be uniquely represented as T Žui . s

n

Ý ai j vj ,

i s 1, 2, . . . , m,

js1

where the a i j ’s are scalars. These scalars completely determine all possible values of T : If u g U, then u s Ý m is1 c i u i for some scalars c1 , c 2 , . . . , c m . Then m n Ž . Ž . T Žu. s Ý m c T u s Ý c Ý is1 i i is1 i js1 a i j vj . By definition, the rectangular array a11 a21 As . . . a m1

a12 a22 . . . am 2

⭈⭈⭈ ⭈⭈⭈ . . . ⭈⭈⭈

a1 n a2 n am n

is called a matrix of order m = n, which indicates that A has m rows and n columns. The a i j ’s are called the elements of A. In some cases it is more convenient to represent A using the notation A s Ž a i j .. In particular, if m s n, then A is called a square matrix. Furthermore, if the off-diagonal elements of a square matrix A are zero, then A is called a diagonal matrix and is written as A s DiagŽ a11 , a22 , . . . , a n n .. In this special case, if the diagonal elements are equal to 1, then A is called the identity matrix and is denoted by I n to indicate that it is of order n = n. A matrix of order m = 1 is called a column vector. Likewise, a matrix of order 1 = n is called a row vector. 2.3.1. Basic Operations on Matrices 1. Equality of Matrices. Let A s Ž a i j . and B s Ž bi j . be two matrices of the same order. Then A s B if and only if a i j s bi j for all i s 1, 2, . . . , m; j s 1, 2, . . . , n.

29

MATRICES AND DETERMINANTS

2. Addition of Matrices. Let A s Ž a i j . and B s Ž bi j . be two matrices of order m = n. Then A q B is a matrix C s Ž c i j . of order m = n such that c i j s a i j q bi j Ž i s 1, 2, . . . , m; j s 1, 2, . . . , n.. 3. Scalar Multiplication. Let ␣ be a scalar, and A s Ž a i j . be a matrix of order m = n. Then ␣ A s Ž ␣ a i j .. 4. The Transpose of a Matrix. Let A s Ž a i j . be a matrix of order m = n. The transpose of A, denoted by A⬘, is a matrix of order n = m whose rows are the columns of A. For example, if

As

2 y1

3 0

1 , 7

then

2 A⬘ s 3 1

y1 0 . 7

A matrix A is symmetric if A s A⬘. It is skew-symmetric if A⬘ s yA. A skew-symmetric matrix must necessarily have zero elements along its diagonal. 5. Product of Matrices. Let A s Ž a i j . and B s Ž bi j . be matrices of orders m = n and n = p, respectively. The product AB is a matrix C s Ž c i j . of order m = p such that c i j s Ý nks1 a i k bk j Ž i s 1, 2, . . . , m; j s 1, 2, . . . , p .. It is to be noted that this product is defined only when the number of columns of A is equal to the number of rows of B. In particular, if a and b are column vectors of order n = 1, then their dot product a ⭈ b can be expressed as a matrix product of the form a⬘b or b⬘a. 6. The Trace of a Matrix. Let A s Ž a i j . be a square matrix of order n = n. The trace of A, denoted by trŽA., is the sum of its diagonal elements, that is, tr Ž A . s

n

Ý aii .

is1

On the basis of this definition, it is easy to show that if A and B are matrices of order n = n, then the following hold: Ži. trŽAB. s trŽBA.; Žii. trŽA q B. s trŽA. q trŽB.. Definition 2.3.1. Let A s Ž a i j . be an m = n matrix. A submatrix B of A is a matrix which can be obtained from A by deleting a certain number of rows and columns. In particular, if the ith row and jth column of A that contain the element a i j are deleted, then the resulting matrix is denoted by M i j Ž i s 1, 2, . . . , m; j s 1, 2, . . . , n.. Let us now suppose that A is a square matrix of order n = n. If rows i1 , i 2 , . . . , i p and columns i1 , i 2 , . . . , i p are deleted from A, where p- n, then the resulting submatrix is called a principal submatrix of A. In particular, if the deleted rows and columns are the last p rows and the last p columns, respectively, then such a submatrix is called a leading principal submatrix.

30

BASIC CONCEPTS IN LINEAR ALGEBRA

Definition 2.3.2. A partitioned matrix is a matrix that consists of several submatrices obtained by drawing horizontal and vertical lines that separate it into groups of rows and columns. For example, the matrix 1 As 6 3

. . 0 . . 2 . . 2 .

3 10 1

. . 4 . . 5 . . 0 .

y5 0 2

is partitioned into six submatrices by drawing one horizontal line and two vertical lines as shown above. Definition 2.3.3. Let A s Ž a i j . be an m1 = n1 matrix and B be an m 2 = n 2 matrix. The direct Žor Kronecker. product of A and B, denoted by A m B, is a matrix of order m1 m 2 = n1 n 2 defined as a partitioned matrix of the form

AmBs

a11 B

a12 B

⭈⭈⭈

a1 n1 B

a21 B . . . a m 11 B

a22 B . . . am 2 2 B

⭈⭈⭈

a2 n1 B . . . . am 1 n1 B

⭈⭈⭈

This matrix can be simplified by writing A m B s w a i j Bx.

I

Properties of the direct product can be found in several matrix algebra books and papers. See, for example, Graybill Ž1983, Section 8.8., Henderson and Searle Ž1981., Magnus and Neudecker Ž1988, Chapter 2., and Searle Ž1982, Section 10.7.. Some of these properties are listed below: 1. 2. 3. 4.

ŽA m B.⬘ s A⬘ m B⬘. A m ŽB m C. s ŽA m B. m C. ŽA m B.ŽC m D. s AC m BD, if AC and BD are defined. trŽA m B. s trŽA.trŽB., if A and B are square matrices.

The paper by Henderson, Pukelsheim, and Searle Ž1983. gives a detailed account of the history associated with direct products. Definition 2.3.4. Let A 1 , A 2 , . . . , A k be matrices of orders m i = n i Ž i s k 1, 2, . . . , k .. The direct sum of these matrices, denoted by [is1 A i , is a k k partitioned matrix of order ŽÝ is1 m i . = ŽÝ is1 n i . that has the block-diagonal form k

[ A i s Diag Ž A 1 , A 2 , . . . , A k . . is1

31

MATRICES AND DETERMINANTS

The following properties can be easily shown on the basis of the preceding definition: 1.

k k k ŽA i q B i ., if A i and B i are of the same order [is1 A i q [is1 B i s [is1

for i s 1, 2, . . . , k. k k k 2. w[is1 A i xw[is1 B i x s [is1 A i B i , if A i B i is defined for i s 1, 2, . . . , k. X k k 3. w[is1 A i x⬘ s [is1 A i . k A i . s Ý kis1 trŽA i .. I 4. trŽ[is1

Definition 2.3.5. Let A s Ž a i j . be a square matrix of order n = n. The determinant of A, denoted by detŽA., is a scalar quantity that can be computed iteratively as det Ž A . s

n

Ý Ž y1. jq1 a1 j det Ž M 1 j . ,

Ž 2.3 .

js1

where M 1 j is a submatrix of A obtained by deleting row 1 and column j Ž j s 1, 2, . . . , n.. For each j, the determinant of M 1 j is obtained in terms of determinants of matrices of order Ž n y 2. = Ž n y 2. using a formula similar to Ž2.3.. This process is repeated several times until the matrices on the right-hand side of Ž2.3. become of order 2 = 2. The determinant of a 2 = 2 matrix such as b s Ž bi j . is given by detŽB. s b11 b 22 y b12 b 21 . Thus by an iterative application of formula Ž2.3., the value of detŽA. can be fully determined. For example, let A be the matrix 1 As 5 1

2 0 2

y1 3 . 1

Then detŽA. s detŽA 1 . y 2 detŽA 2 . y detŽA 3 ., where A 1 , A 2 , A 3 are 2 = 2 submatrices, namely A1 s

0 2

3 , 1

A2s

5 1

3 , 1

It follows that detŽA. s y6 y 2Ž2. y 10 s y20.

A3s

5 1

0 . 2

I

Definition 2.3.6. Let A s Ž a i j . be a square matrix order of n = n. The determinant of M i j , the submatrix obtained by deleting row i and column j, is called a minor of A of order n y 1. The quantity Žy1. iqj detŽM i j . is called a cofactor of the corresponding Ž i, j .th element of A. More generally, if A is an m = n matrix and if we strike out all but p rows and the same number of columns from A, where pF minŽ m, n., then the determinant of the resulting submatrix is called a minor of A of order p.

32

BASIC CONCEPTS IN LINEAR ALGEBRA

The determinant of a principal submatrix of a square matrix A is called a principal minor. If, however, we have a leading principal submatrix, then its determinant is called a leading principal minor. I NOTE 2.3.1. square matrix.

The determinant of a matrix A is defined only when A is a

NOTE 2.3.2. The expansion of detŽA. in Ž2.3. was carried out by multiplying the elements of the first row of A by their corresponding cofactors and then summing over j Žs 1, 2, . . . , n.. The same value of detŽA. could have also been obtained by similar expansions according to the elements of any row of A Žinstead of the first row., or any column of A. Thus if M i j is a submatrix of A obtained by deleting row i and column j, then detŽA. can be obtained by using any of the following expansions: By row i:

det Ž A . s

n

Ý Ž y1. iqj ai j det Ž M i j . ,

i s 1, 2, . . . , n.

js1

By column j:

det Ž A . s

n

Ý Ž y1. iqj ai j det ŽM i j . ,

j s 1, 2, . . . , n.

is1

NOTE 2.3.3.

Some of the properties of determinants are the following:

detŽAB. s detŽA.detŽB., if A and B are n = n matrices. If A⬘ is the transpose of A, then detŽA⬘. s detŽA.. If A is an n = n matrix and ␣ is a scalar, then detŽ ␣ A. s ␣ n detŽA.. If any two rows Žor columns. of A are identical, then detŽA. s 0. If any two rows Žor columns. of A are interchanged, then detŽA. is multiplied by y1. vi. If detŽA. s 0, then A is called a singular matrix. Otherwise, A is a nonsingular matrix. vii. If A and B are matrices of orders m = m and n = n, respectively, then the following hold: Ža. detŽA m B. s wdetŽA.x nwdetŽB.x m ; Žb. detŽA [ B. s wdetŽA.xwdetŽB.x. i. ii. iii. iv. v.

NOTE 2.3.4. The history of determinants dates back to the fourteenth century. According to Smith Ž1958, page 273., the Chinese had some knowledge of determinants as early as about 1300 A.D. Smith Ž1958, page 440. also Ž1642᎐1708. had reported that the Japanese mathematician Seki Kowa ˜ discovered the expansion of a determinant in solving simultaneous equations. In the West, the theory of determinants is believed to have originated with the German mathematician Gottfried Leibniz Ž1646᎐1716. in 1693, ten years

33

MATRICES AND DETERMINANTS

after the work of Seki Kowa. ˜ However, the actual development of the theory of determinants did not begin until the publication of a book by Gabriel Cramer Ž1704᎐1752. Žsee Price, 1947, page 85. in 1750. Other mathematicians who contributed to this theory include Alexandre Vandermonde Ž1735᎐1796., Pierre-Simon Laplace Ž1749᎐1827., Carl Gauss Ž1777᎐1855., and Augustin-Louis Cauchy Ž1789᎐1857.. Arthur Cayley Ž1821᎐1895. is credited with having been the first to introduce the common present-day notation of vertical bars enclosing a square matrix. For more interesting facts about the history of determinants, the reader is advised to read the article by Price Ž1947.. 2.3.2. The Rank of a Matrix Let A s Ž a i j . be a matrix of order m = n. Let uX1 , uX2 , . . . , uXm denote the row vectors of A, and let v1 , v2 , . . . , vn denote its column vectors. Consider the linear spans of the row and column vectors, namely, V1 s LŽuX1 ,uX2 , . . . , uXm ., V2 s LŽv1 , v2 , . . . , vn ., respectively. Theorem 2.3.1.

The vector spaces V1 and V2 have the same dimension.

Proof. See Lancaster Ž1969, Theorem 1.15.1., or Searle Ž1982, Section 6.6.. I Thus, for any matrix A, the number of linearly independent rows is the same as the number of linearly independent columns. Definition 2.3.7. The rank of a matrix A is the number of its linearly independent rows Žor columns.. The rank of A is denoted by r ŽA.. I Theorem 2.3.2. If a matrix A has a nonzero minor of order r, and if all minors of order r q 1 and higher Žif they exist. are zero, then A has rank r. Proof. See Lancaster Ž1969, Lemma 1, Section 1.15..

I

For example, if A is the matrix 2 As 0 2

3 1 4

y1 2 , 1

then r ŽA. s 2. This is because detŽA. s 0 and at least one minor of order 2 is different from zero.

34

BASIC CONCEPTS IN LINEAR ALGEBRA

There are several properties associated with the rank of a matrix. Some of these properties are the following: 1. r ŽA. s r ŽA⬘.. 2. The rank of A is unchanged if A is multiplied by a nonsingular matrix. Thus if A is an m = n matrix and P is an n = n nonsingular matrix, then r ŽA. s r ŽAP.. 3. r ŽA. s r ŽAA⬘. s r ŽA⬘A.. 4. If the matrix A is partitioned as A s wA 1 : A 2 x, where A 1 and A 2 are submatrices of the same order, then r ŽA 1 q A 2 . F r ŽA. F r ŽA 1 . q r ŽA 2 .. More generally, if the matrices A 1 , A 2 , . . . , A k are of the same order and if A is partitioned as A s wA 1 : A 2 : ⭈⭈⭈ : A k x, then

ž / k

r

Ý Ai

F r Ž A. F

is1

k

Ý r ŽA i . .

is1

5. If the product AB is defined, then r ŽA. q r ŽB. y n F r ŽAB. F min
r ŽA., r ŽB.4 , where n is the number of columns of A Žor the number of rows of B.. 6. r ŽA m B. s r ŽA. r ŽB.. 7. r ŽA [ B. s r ŽA. q r ŽB.. Definition 2.3.8. Let A be a matrix of order m = n and rank r. Then we have the following: 1. A is said to have a full row rank if r s m - n. 2. A is said to have a full column rank if r s n - m. 3. A is of full rank if r s m s n. In this case, detŽA. / 0, that is, A is a nonsingular matrix. I 2.3.3. The Inverse of a Matrix Let A s Ž a i j . be a nonsingular matrix of order n = n. The inverse of A, denoted by Ay1 , is an n = n matrix that satisfies the condition AAy1 s Ay1A s I n. The inverse of A can be computed as follows: Let c i j be the cofactor of a i j Žsee Definition 2.3.6.. Define the matrix C as C s Ž c i j .. The transpose of C is called the adjugate or adjoint of A and is denoted by adj A. The inverse of A is then given by Ay1 s

adj A det Ž A .

.

35

MATRICES AND DETERMINANTS

It can be verified that A

adj A

s

det Ž A .

adj A det Ž A .

AsIn.

For example, if A is the matrix 2 A s y3 2

0 2 1

2 3 y7

1 0 y2

1 0 , 1

then detŽA. s y3, and

adj A s

y2 y3 . 4

Hence, y 23

y 13

Ay1 s y1

0

7 3

2 3

2 3

1 . y 43

Some properties of the inverse operation are given below: 1. 2. 3. 4. 5. 6. 7.

ŽAB.y1 s By1Ay1 . ŽA⬘.y1 s ŽAy1 .⬘. detŽAy1 . s 1rdetŽA.. ŽAy1 .y1 s A. ŽA m B.y1 s Ay1 m By1 . ŽA [ B.y1 s Ay1 [ By1 . If A is partitioned as As

A 11 A 21

A 12 , A 22

where A i j is of order n i = n j Ž i, j s 1, 2., then det Ž A . s

½

det Ž A 11 . ⭈ det Ž A 22 y A 21 Ay1 11 A 12 .

det Ž A 22 . ⭈ det Ž

A 11 y A 12 Ay1 22 A 21

.

if A 11 is nonsingular, if A 22 is nonsingular.

36

BASIC CONCEPTS IN LINEAR ALGEBRA

The inverse of A is partitioned as Ay1 s

B 11 B 21

B 12 , B 22

where B 11 s Ž A 11 y A 12 Ay1 22 A 21 .

y1

,

B 12 s yB 11 A 12 Ay1 22 , B 21 s yAy1 22 A 21 B 11 , y1 y1 B 22 s Ay1 22 q A 22 A 21 B 11 A 12 A 22 .

2.3.4. Generalized Inverse of a Matrix This inverse represents a more general concept than the one discussed in the previous section. Let A be a matrix of order m = n. Then, a generalized inverse of A, denoted by Ay, is a matrix of order n = m that satisfies the condition

Ž 2.4 .

AAyA s A.

Note that Ay is defined even if A is not a square matrix. If A is a square matrix, it does not have to be nonsingular. Furthermore, condition Ž2.4. can be satisfied by infinitely many matrices Žsee, for example, Searle, 1982, Chapter 8.. If A is nonsingular, then Ž2.4. is satisfied by only Ay1 . Thus Ay1 is a special case of Ay. Theorem 2.3.3. 1. If A is a symmetric matrix, then Ay can be chosen to be symmetric. 2. AŽA⬘A.y A⬘As A for any matrix A. 3. AŽA⬘A.y A⬘ is invariant to the choice of a generalized inverse of A⬘A. Proof. See Searle Ž1982, pages 221᎐222..

I

2.3.5. Eigenvalues and Eigenvectors of a Matrix Let A be a square matrix of order n = n. By definition, a scalar ␭ is said to be an eigenvalue Žor characteristic root. of A if A y ␭I n is a singular matrix, that is, det Ž A y ␭I n . s 0.

Ž 2.5 .

37

MATRICES AND DETERMINANTS

Thus an eigenvalue of A satisfies a polynomial equation of degree n called the characteristic equation of A. If ␭ is a multiple solution Žor root. of equation Ž2.5., that is, Ž2.5. has several roots, say m, that are equal to ␭, then ␭ is said to be an eigenvalue of multiplicity m. Since r ŽA y ␭I n . - n by the fact that A y ␭I n is singular, the columns of A y ␭I n must be linearly related. Hence, there exists a nonzero vector v such that

Ž A y ␭I n . v s 0,

Ž 2.6 .

Av s ␭v.

Ž 2.7 .

or equivalently,

A vector satisfying Ž2.7. is called an eigenvector Žor a characteristic vector. corresponding to the eigenvalue ␭. From Ž2.7. we note that the linear transformation of v by the matrix A is a scalar multiple of v. The following theorems describe certain properties associated with eigenvalues and eigenvectors. The proofs of these theorems can be found in standard matrix algebra books Žsee the annotated bibliography.. Theorem 2.3.4. A square matrix A is singular if and only if at least one of its eigenvalues is equal to zero. In particular, if A is symmetric, then its rank is equal to the number of its nonzero eigenvalues. Theorem 2.3.5.

The eigenvalues of a symmetric matrix are real.

Theorem 2.3.6. Let A be a square matrix, and let ␭1 , ␭2 , . . . , ␭ k denote its distinct eigenvalues. If v1 , v2 , . . . , vk are eigenvectors of A corresponding to ␭1 , ␭2 , . . . , ␭ k , respectively, then v1 , v2 , . . . , vk are linearly independent. In particular, if A is symmetric, then v1 , v2 , . . . , vk are orthogonal to one another, that is, viX vj s 0 for i / j Ž i, j s 1, 2, . . . , k .. Theorem 2.3.7. Let A and B be two matrices of orders m = m and n = n, respectively. Let ␭1 , ␭2 , . . . , ␭ m be the eigenvalues of A, and ®1 , ®2 , . . . , ®n be the eigenvalues of B. Then we have the following: 1. The eigenvalues of A m B are of the form ␭ i ␯ j Ž i s 1, 2, . . . , m; j s 1, 2, . . . , n.. 2. The eigenvalues of A [ B are ␭1 , ␭2 , . . . , ␭ m ; ␯ 1 , ␯ 2 , . . . , ␯n . Theorem 2.3.8. Let ␭1 , ␭2 , . . . , ␭ n be the eigenvalues of a matrix A of order n = n. Then the following hold: 1. trŽA. s Ý nis1 ␭ i . n 2. detŽA. s Ł is1 ␭i.

38

BASIC CONCEPTS IN LINEAR ALGEBRA

Theorem 2.3.9. Let A and B be two matrices of orders m = n and n = m Ž n G m., respectively. The nonzero eigenvalues of BA are the same as those of AB. 2.3.6. Some Special Matrices 1. The vector 1 n is a column vector of ones of order n = 1. 2. The matrix J n is a matrix of ones of order n = n. 3. Idempotent Matrix. A square matrix A for which A2 s A is called an idempotent matrix. For example, the matrix A s I n y Ž1rn.J n is idempotent of order n = n. The eigenvalues of an idempotent matrix are equal to zeros and ones. It follows from Theorem 2.3.8 that the rank of an idempotent matrix, which is the same as the number of eigenvalues that are equal to 1, is also equal to its trace. Idempotent matrices are used in many applications in statistics Žsee Section 2.4.. 4. Orthogonal Matrix. A square matrix A is orthogonal if A⬘As I. From this definition it follows that Ži. A is orthogonal if and only if A⬘ s Ay1 ; Žii. < detŽA.< s 1. A special orthogonal matrix is the Householder matrix, which is a symmetric matrix of the form H s I y 2uu⬘ru⬘u, where u is a nonzero vector. Orthogonal matrices occur in many applications of matrix algebra and play an important role in statistics, as will be seen in Section 2.4. 2.3.7. The Diagonalization of a Matrix Theorem 2.3.10 ŽThe Spectral Decomposition Theorem.. Let A be a symmetric matrix of order n = n. There exists an orthogonal matrix P such that A s P⌳ P⬘, where ⌳ s DiagŽ ␭1 , ␭ 2 , . . . , ␭ n . is a diagonal matrix whose diagonal elements are the eigenvalues of A. The columns of P are the corresponding orthonormal eigenvectors of A. Proof. See Basilevsky Ž1983, Theorem 5.8, page 200..

I

If P is partitioned as P s wp 1: p 2 : ⭈⭈⭈ :p n x, where p i is an eigenvector of A with eigenvalue ␭ i Ž i s 1, 2, . . . , n., then A can be written as n

As

Ý ␭i p i pXi .

is1

39

MATRICES AND DETERMINANTS

For example, if As

1 0 y2

y2 0 , 4

0 0 0

then A has two distinct eigenvalues, ␭1 s 0 of multiplicity 2 and ␭2 s 5. For ␭1 s 0 we have two orthonormal eigenvectors, p 1 s Ž2, 0, 1.⬘r '5 and p 2 s Ž0, 1, 0.⬘. Note that p 1 and p 2 span the kernel Žnull space. of the linear transformation represented by A. For ␭2 s 5 we have the normal eigenvector p 3 s Ž1, 0,y 2.⬘r '5 , which is orthogonal to both p 1 and p 2 . Hence, P and ⌳ in Theorem 2.3.10 for the matrix A are 2

Ps

1

'5

0

'5

0 1

1

0 y2

0

'5

'5

,

⌳ s Diag Ž 0, 0, 5 . . The next theorem gives a more general form of the spectral decomposition theorem. Theorem 2.3.11 ŽThe Singular-Value Decomposition Theorem.. Let A be a matrix of order m = n Ž m F n. and rank r. There exist orthogonal matrices P and Q such that A s PwD : 0xQ⬘, where D s DiagŽ ␭1 , ␭2 , . . . , ␭ m . is a diagonal matrix with nonnegative diagonal elements called the singular values of A, and 0 is a zero matrix of order m = Ž n y m.. The diagonal elements of D are the square roots of the eigenvalues of AA⬘. Proof. See, for example, Searle Ž1982, pages 316᎐317..

I

2.3.8. Quadratic Forms Let A s Ž a i j . be a symmetric matrix of order n = n, and let x s Ž x 1 , x 2 , . . . , x n .⬘ be a column vector of order n = 1. The function q Ž x . s x⬘Ax n

s

n

Ý Ý ai j x i x j

is1 js1

is called a quadratic form in x.

40

BASIC CONCEPTS IN LINEAR ALGEBRA

A quadratic form x⬘Ax is said to be the following: 1. Positive definite if x⬘Ax ) 0 for all x / 0 and is zero only if x s 0. 2. Positive semidefinite if x⬘Ax G 0 for all x and x⬘Ax s 0 for at least one nonzero value of x. 3. Nonnegative definite if A is either positive definite or positive semidefinite. Theorem 2.3.12. Let A s Ž a i j . be a symmetric matrix of order n = n. Then A is positive definite if and only if either of the following two conditions is satisfied: 1. The eigenvalues of A are all positive. 2. The leading principal minors of A are all positive, that is,

a11 ) 0,

det

ž

a11 a21

a12 a22

/

) 0, . . . ,

det Ž A . ) 0.

Proof. The proof of part 1 follows directly from the spectral decomposition theorem. For the proof of part 2, see Lancaster Ž1969, Theorem 2.14.4.. I Theorem 2.3.13. Let A s Ž a i j . be a symmetric matrix of order n = n. Then A is positive semidefinite if and only if its eigenvalues are nonnegative with at least one of them equal to zero. Proof. See Basilevsky Ž1983, Theorem 5.10, page 203..

I

2.3.9. The Simultaneous Diagonalization of Matrices By simultaneous diagonalization we mean finding a matrix, say Q, that can reduce several square matrices to a diagonal form. In many situations there may be a need to diagonalize several matrices simultaneously. This occurs frequently in statistics, particularly in analysis of variance. The proofs of the following theorems can be found in Graybill Ž1983, Chapter 12.. Theorem 2.3.14.

Let A and B be symmetric matrices of order n = n.

1. If A is positive definite, then there exists a nonsingular matrix Q such that Q⬘AQs I n and Q⬘BQs D, where D is a diagonal matrix whose diagonal elements are the roots of the polynomial equation detŽB y ␭ A. s 0.

41

MATRICES AND DETERMINANTS

2. If A and B are positive semidefinite, then there exists a nonsingular matrix Q such that Q⬘AQs D 1 , Q⬘BQs D 2 , where D 1 and D 2 are diagonal matrices Žfor a detailed proof of this result, see Newcomb, 1960.. Theorem 2.3.15. Let A 1 , A 2 , . . . , A k be symmetric matrices of order n = n. Then there exists an orthogonal matrix P such that A i s P⌳ i P⬘,

i s 1, 2, . . . , k,

where ⌳ i is a diagonal matrix, if and only if A i A j s A j A i for all i / j Ž i, j s 1, 2, . . . , k .. 2.3.10. Bounds on Eigenvalues Let A be a symmetric matrix of order n = n. We denote the ith eigenvalue of A by e i ŽA., i s 1, 2, . . . , n. The smallest and largest eigenvalues of A are denoted by emin ŽA. and emax ŽA., respectively. Theorem 2.3.16.

emin ŽA. F x⬘Axrx⬘x F emax ŽA..

Proof. This follows directly from the spectral decomposition theorem.

I

The ratio x⬘Axrx⬘x is called Rayleigh’s quotient for A. The lower and upper bounds in Theorem 2.3.16 can be achieved by choosing x to be an eigenvector associated with emin ŽA. and emax ŽA., respectively. Thus Theorem 2.3.16 implies that inf x/0

sup x/0

x⬘Ax x⬘x x⬘Ax x⬘x

s emin Ž A . ,

Ž 2.8 .

s emax Ž A . .

Ž 2.9 .

Theorem 2.3.17. If A is a symmetric matrix and B is a positive definite matrix, both of order n = n, then emin Ž By1A . F

x⬘Ax x⬘Bx

Proof. The proof is left to the reader.

F emax Ž By1A . I

42

BASIC CONCEPTS IN LINEAR ALGEBRA

Note that the above lower and upper bounds are equal to the infimum and supremum, respectively, of the ratio x⬘Axrx⬘Bx for x / 0. Theorem 2.3.18. If A is a positive semidefinite matrix and B is a positive definite matrix, both of order n = n, then for any i Ž i s 1, 2, . . . , n., e i Ž A . emin Ž B . F e i Ž AB . F e i Ž A . emax Ž B . .

Ž 2.10 .

Furthermore, if A is positive definite, then for any i Ž i s 1, 2, . . . , n., e i2 Ž AB . emax Ž A . emax Ž B .

e i2 Ž AB .

F ei Ž A. ei Ž B. F

emin Ž A . emin Ž B .

Proof. See Anderson and Gupta Ž1963, Corollary 2.2.1..

I

A special case of the double inequality in Ž2.10. is emin Ž A . emin Ž B . F e i Ž AB . F emax Ž A . emax Ž B . , for all i Ž i s 1, 2, . . . , n.. Theorem 2.3.19. Let A and B be symmetric matrices of order n = n. Then, the following hold: 1. e i ŽA. F e i ŽA q B., i s 1, 2, . . . , n, if B is nonnegative definite. 2. e i ŽA. - e i ŽA q B., i s 1, 2, . . . , n, if B is positive definite. Proof. See Bellman Ž1970, Theorem 3, page 117..

I

Theorem 2.3.20 ŽSchur’s Theorem.. Let A s Ž a i j . be a symmetric matrix of order n = n, and let 5 A 5 2 denote its Euclidean norm, defined as 5 A52s

žÝ Ý / n

n

1r2

a2i j

.

is1 js1

Then n

Ý ei2 Ž A. s 5 A 5 22 .

is1

Proof. See Lancaster Ž1969, Theorem 7.3.1..

I

Since 5 A 5 2 F n max i, j < a i j < , then from Theorem 2.3.20 we conclude that < emax Ž A . < F n max < a i j < . i, j

43

APPLICATIONS OF MATRICES IN STATISTICS

Theorem 2.3.21. Let A be a symmetric matrix of order n = n, and let m and s be defined as ms

tr Ž A .

,

n

ss

ž

tr Ž A2 . n

y m2

/

1r2

.

Then m y s Ž n y 1. mq

1r2

s

Ž n y 1.

1r2

F emin Ž A . F m y

s

Ž n y 1.

1r2

F emax Ž A . F m q s Ž n y 1 .

emax Ž A . y emin Ž A . F s Ž 2 n .

1r2

1r2

, ,

.

Proof. See Wolkowicz and Styan Ž1980, Theorems 2.1 and 2.5..

I

2.4. APPLICATIONS OF MATRICES IN STATISTICS The use of matrix algebra is quite prevalent in statistics. In fact, in the areas of experimental design, linear models, and multivariate analysis, matrix algebra is considered the most frequently used branch of mathematics. Applications of matrices in these areas are well documented in several books, for example, Basilevsky Ž1983., Graybill Ž1983., Magnus and Neudecker Ž1988., and Searle Ž1982.. We shall therefore not attempt to duplicate the material given in these books. Let us consider the following applications: 2.4.1. The Analysis of the Balanced Mixed Model In analysis of variance, a linear model associated with a given experimental situation is said to be balanced if the numbers of observations in the subclasses of the data are the same. For example, the two-way crossed-classification model with interaction, yi jk s ␮ q ␣ i q ␤ j q Ž ␣␤ . i j q ⑀ i jk ,

Ž 2.11 .

i s 1, 2, . . . , a; j s 1, 2, . . . , b; k s 1, 2, . . . , n, is balanced, since there are n observations for each combination of i and j. Here, ␣ i and ␤ j represent the main effects of the factors under consideration, Ž ␣␤ . i j denotes the interaction effect, and ⑀ i jk is a random error term. Model Ž2.11. can be written in vector form as y s H 0␶ 0 q H 1␶ 1 q H 2 ␶ 2 q H 3 ␶ 3 q H 4 ␶4 ,

Ž 2.12 .

44

BASIC CONCEPTS IN LINEAR ALGEBRA

where y is the vector of observations, ␶ 0 s ␮ , ␶ 1 s Ž ␣ 1 , ␣ 2 , . . . , ␣ a .⬘, ␶ 2 s Ž ␤ 1 , ␤ 2 , . . . , ␤ b .⬘, ␶ 3 s wŽ ␣ ␤ . 11 , Ž ␣ ␤ . 12 , . . . , Ž ␣ ␤ . a b x⬘, and ␶4 s Ž ⑀ 111 , ⑀ 112 , . . . , ⑀ ab n .⬘. The matrices H i Ž i s 0, 1, 2, 3, 4. can be expressed as direct products of the form H0 s1a m1b m1n , H1 s I a m 1 b m 1 n , H2 s1a mIb m1n , H3 sI a mIb m1n , H4 sI a mIb mIn. In general, any balanced linear model can be written in vector form as ␯

Ý H l ␶l ,

ys

Ž 2.13 .

ls0

where H l Ž l s 0, 1, . . . , ␯ . is a direct product of identity matrices and vectors of ones Žsee Khuri, 1982.. If ␶ 0 , ␶ 1 , . . . , ␶␪ Ž ␪ - ␯ y 1. are fixed unknown parameter vectors Žfixed effects ., and ␶␪q1 , ␶␪q2 , . . . , ␶␯ are random vectors Žrandom effects ., then model Ž2.11. is called a balanced mixed model. Furthermore, if we assume that the random effects are independent and have the normal distributions N Ž0, ␴ l 2 I c l ., where c l is the number of columns of H l , l s ␪ q 1, ␪ q 2, . . . , ␯ , then, because model Ž2.11. is balanced, its statistical analysis becomes very simple. Here, the ␴ l 2 ’s are called the model’s variance components. A balanced mixed model can be written as y s Xg q Zh

Ž 2.14 .

where Xg s Ý␪ls0 H l ␶ l is the fixed portion of the model, and Zh s Ý␯ls␪q1 H l ␶ l is its random portion. The variance᎐covariance matrix of y is given by ⌺s

␯

Ý

ls␪q1

A l ␴l2 ,

where A l s H l HXl Ž l s ␪ q 1, ␪ q 2, . . . , ␯ .. Note that A l A p s A p A l for all l / p. Hence, the matrices A l can be diagonalized simultaneously Žsee Theorem 2.3.15.. If y⬘Ay is a quadratic form in y, then y⬘Ay is distributed as a noncentral chi-squared variate ␹mX 2 Ž␩ . if and only if A ⌺ is idempotent of rank m, where ␩ is the noncentrality parameter and is given by ␩ s g⬘X⬘AXg Žsee Searle, 1971, Section 2.5.. The total sum of squares, y⬘y, can be uniquely partitioned as ␯

y⬘ys

Ý y⬘Pl y,

ls0

45

APPLICATIONS OF MATRICES IN STATISTICS

where the Pl ’s are idempotent matrices such that Pl Ps s 0 for all l / s Žsee Khuri, 1982.. The quadratic form y⬘Pl y Ž l s 0, 1, . . . , ␯ . is positive semidefinite and represents the sum of squares for the lth effect in model Ž2.13.. Theorem 2.4.1. Consider the balanced mixed model Ž2.14., where the random effects are assumed to be independently and normally distributed with zero means and variance᎐covariance matrices ␴ l 2 I c l Ž l s ␪ q 1, ␪ q 2, . . . , ␯ .. Then we have the following: 1. y⬘P0 y, y⬘P1 y, . . . , y⬘P␯ y are statistically independent. 2. y⬘Pl yr␦ l is distributed as a noncentral chi-squared variate with degrees of freedom equal to the rank of Pl and noncentrality parameter given by ␩l s g⬘X⬘Pl Xgr␦ l for l s 0, 1, . . . , ␪ , where ␦ l is a particular linear 2 2 combination of the variance components ␴␪q1 , ␴␪q2 , . . . , ␴␯ 2 . However, for l s ␪ q 1, ␪ q 2, . . . , ␯ , that is, for the random effects, y⬘Pl yr␦ l is distributed as a central chi-squared variate with m l degrees of freedom, where m l s r ŽPl .. Proof. See Theorem 4.1 in Khuri Ž1982..

I

Theorem 2.4.1 provides the basis for a complete analysis of any balanced mixed model, as it can be used to obtain exact tests for testing the significance of the fixed effects and the variance components. A linear function a⬘g, of g in model Ž2.14., is estimable if there exists a linear function, c⬘y, of the observations such that E Žc⬘y. s a⬘g. In Searle Ž1971, Section 5.4. it is shown that a⬘g is estimable if and only if a⬘ belongs to the linear span of the rows of X. In Khuri Ž1984. we have the following theorem: Theorem 2.4.2. Consider the balanced mixed model in Ž2.14.. Then we have the following: 1. r ŽPl X. s r ŽPl ., l s 0, 1, . . . , ␪ . 2. r ŽX. s Ý␪ls0 r ŽPl X.. 3. P0 Xg, P1 Xg, . . . , P␪ Xg are linearly independent and span the space of all estimable linear functions of g. Theorem 2.4.2 is useful in identifying a basis of estimable linear functions of the fixed effects in model Ž2.14.. 2.4.2. The Singular-Value Decomposition The singular-value decomposition of a matrix is far more useful, both in statistics and in matrix algebra, then is commonly realized. For example, it

46

BASIC CONCEPTS IN LINEAR ALGEBRA

plays a significant role in regression analysis. Let us consider the linear model y s X␤ q ⑀ ,

Ž 2.15 .

where y is a vector of n observations, X is an n = p Ž n G p . matrix consisting of known constants, ␤ is an unknown parameter vector, and ⑀ is a random error vector. Using Theorem 2.3.11, the matrix X⬘ can be expressed as X⬘ s P w D : 0 x Q⬘,

Ž 2.16 .

where P and Q are orthogonal matrices of orders p= p and n = n, respectively, and D is a diagonal matrix of order p= p consisting of nonnegative diagonal elements. These are the singular values of X Žor of X⬘. and are the positive square roots of the eigenvalues of X⬘X. From Ž2.16. we get XsQ

D P⬘. 0⬘

Ž 2.17 .

If the columns of X are linearly related, then they are said to be multicollinear. In this case, X has rank r Ž- p ., and the columns of X belong to a vector subspace of dimension r. At least one of the eigenvalues of X⬘X, and hence at least one of the singular values of X, will be equal to zero. In practice, such exact multicollinearities rarely occur in statistical applications. Rather, the columns of X may be ‘‘nearly’’ linearly related. In this case, the rank of X is p, but some of the singular values of X will be ‘‘near zero.’’ We shall use the term multicollinearity in a broader sense to describe the latter situation. It is also common to use the term ‘‘ill conditioning’’ to refer to the same situation. The presence of multicollinearities in X can have adverse effects on the ˆ of ␤ in Ž2.15.. This can be easily seen from the fact least-squares estimate, ␤, ˆ s ŽX⬘X.y1 X⬘y and VarŽ␤ ˆ . s ŽX⬘X.y1␴ 2 , where ␴ 2 is the error varithat ␤ ˆ can therefore be ance. Large variances associated with the elements of ␤ ˆ to become expected when the columns of X are multicollinear. This causes ␤ an unreliable estimate of ␤. For a detailed study of multicollinearity and its effects, see Belsley, Kuh, and Welsch Ž1980, Chapter 3., Montgomery and Peck Ž1982, Chapter 8., and Myers Ž1990, Chapter 3.. The singular-value decomposition of X can provide useful information for detecting multicollinearity, as we shall now see. Let us suppose that the columns of X are multicollinear. Because of this, some of the singular values of X, say p 2 Ž- p . of them, will be ‘‘near zero.’’ Let us partition D in Ž2.17. as Ds

D1 0

0 , D2

47

APPLICATIONS OF MATRICES IN STATISTICS

where D 1 and D 2 are of orders p1 = p1 and p 2 = p 2 Ž p1 s py p 2 ., respectively. The diagonal elements of D 2 consist of those singular values of X labeled as ‘‘near zero.’’ Let us now write Ž2.17. as D1 XP s Q 0 0

0 D2 . 0

Ž 2.18 .

Let us next partition P and Q as P s wP1 : P2 x, Q s wQ 1: Q 2 x, where P1 and P2 have p1 and p 2 columns, respectively, and Q 1 and Q 2 have p1 and n y p1 columns, respectively. From Ž2.18. we conclude that XP1 s Q 1 D 1 ,

Ž 2.19 .

XP2 f 0,

Ž 2.20 .

where f represents approximate equality. The matrix XP2 is ‘‘near zero’’ because of the smallness of the diagonal elements of D 2 . We note from Ž2.20. that each column of P2 provides a ‘‘near’’-linear relationship among the columns of X. If Ž2.20. were an exact equality, then the columns of P2 would provide an orthonormal basis for the null space of X. We have mentioned that the presence of multicollinearity is indicated by the ‘‘smallness’’ of the singular values of X. The problem now is to determine what ‘‘small’’ is. For this purpose it is common in statistics to use the condition number of X, denoted by ␬ ŽX.. By definition

␬ Ž X. s

␭max ␭min

,

where ␭max and ␭min are, respectively, the largest and smallest singular values of X. Since the singular values of X are the positive square roots of the eigenvalues of X⬘X, then ␬ ŽX. can also be written as

␬ Ž X. s

(

emax Ž X⬘X . emin Ž X⬘X .

.

If ␬ ŽX. is less than 10, then there is no serious problem with multicollinearity. Values of ␬ ŽX. between 10 and 30 indicate moderate to strong multicollinearity, and if ␬ ) 30, severe multicollinearity is implied. More detailed discussions concerning the use of the singular-value decomposition in regression can be found in Mandel Ž1982.. See also Lowerre Ž1982.. Good Ž1969. described several applications of this decomposition in statistics and in matrix algebra.

48

BASIC CONCEPTS IN LINEAR ALGEBRA

2.4.3. Extrema of Quadratic Forms In many statistical problems there is a need to find the extremum Žmaximum or minimum. of a quadratic form or a ratio of quadratic forms. Let us, for example, consider the following problem: Let X 1 , X 2 , . . . , X n be a collection of random vectors, all having the same number of elements. Suppose that these vectors are independently and identically distributed Ži.i.d. . as N Ž␮, ⌺ ., where both ␮ and ⌺ are unknown. Consider testing the hypothesis H0 : ␮ s ␮ 0 versus its alternative Ha : ␮ / ␮ 0 , where ␮ 0 is some hypothesized value of ␮. We need to develop a test statistic for testing H0 . The multivariate hypothesis H0 is true if and only if the univariate hypotheses H0 Ž ␭ . : ␭⬘␮ s ␭⬘␮ 0 are true for all ␭ / 0. A test statistic for testing H0 Ž ␭ . is the following: tŽ ␭ . s

␭⬘ Ž X y ␮ 0 . 'n

'␭⬘S␭

,

where X s Ý nis1 X irn and S is the sample variance᎐covariance matrix, which is an unbiased estimator of ⌺, and is given by Ss

n

1 ny1

Ý Ž X i y X .Ž X i y X . ⬘.

is1

Large values of t 2 Ž ␭ . indicate falsehood of H0 Ž ␭ .. Since H0 is rejected if and only if H0 Ž ␭ . is rejected for at least one ␭ , then the condition to reject H0 at the ␣-level is sup ␭ / 0 w t 2 Ž ␭ .x ) c␣ , where c␣ is the upper 100 ␣ % point of the distribution of sup ␭ / 0 w t 2 Ž ␭ .x. But sup t Ž ␭ . s sup 2

␭/0

n < ␭⬘ Ž X y ␮ 0 . < 2

␭/0

s n sup

␭⬘S␭ ␭⬘ Ž X y ␮ 0 .Ž X y ␮ 0 . ⬘␭ ␭⬘S␭

␭/0

s n emax Sy1 Ž X y ␮ 0 .Ž X y ␮ 0 . ⬘ , by Theorem 2.3.17. Now, emax Sy1 Ž X y ␮ 0 .Ž X y ␮ 0 . ⬘ s emax

Ž X y ␮ 0 . ⬘Sy1 Ž X y ␮ 0 . s Ž X y ␮ 0 . ⬘Sy1 Ž X y ␮ 0 . .

,

by Theorem 2.3.9.

49

APPLICATIONS OF MATRICES IN STATISTICS

Hence, sup t 2 Ž ␭ . s n Ž X y ␮ 0 . ⬘Sy1 Ž X y ␮ 0 .

␭/0

is the test statistic for the multivariate hypothesis H0 . This is called Hotelling’s T 2-statistic. Its critical values are obtained in terms of the critical values of the F-distribution Žsee, for example, Morrison, 1967, Chapter 4.. Another example of using the extremum of a ratio of quadratic forms is in the determination of the canonical correlation coefficient between two random vectors Žsee Exercise 2.26.. The article by Bush and Olkin Ž1959. lists several similar statistical applications. 2.4.4. The Parameterization of Orthogonal Matrices Orthogonal matrices are used frequently in statistics, especially in linear models and multivariate analysis Žsee, for example, Graybill, 1961, Chapter 11; James, 1954.. The n2 elements of an n = n orthogonal matrix Q are subject to nŽ n q 1.r2 constraints because Q⬘Qs I n . These elements can therefore be represented by n2 y nŽ n q 1.r2 s nŽ n y 1.r2 independent parameters. The need for such a representation arises in several situations. For example, in the design of experiments, there may be a need to search for an orthogonal matrix that satisfies a certain optimality criterion. Using the independent parameters of an orthogonal matrix can facilitate this search. Khuri and Myers Ž1981. followed this approach in their construction of a response surface design that is robust to nonnormality of the error distribution associated with the response function. Another example is the generation of random orthogonal matrices for carrying out simulation experiments. This was used by Heiberger, Velleman, and Ypelaar Ž1983. to construct test data with special properties for multivariate linear models. Anderson, Olkin, and Underhill Ž1987. proposed a procedure to generate random orthogonal matrices. Methods to parameterize an orthogonal matrix were reviewed in Khuri and Good Ž1989.. One such method is to use the relationship between an orthogonal matrix and a skew-symmetric matrix. If Q is an orthogonal matrix with determinant equal to one, then it can be written in the form Q s eT , where T is a skew-symmetric matrix Žsee, for example, Gantmacher, 1959.. The elements of T above its main diagonal can be used to parameterize Q. This exponential mapping is defined by the infinite series eT s I q T q

T2 2!

q

T3 3!

q ⭈⭈⭈ .

50

BASIC CONCEPTS IN LINEAR ALGEBRA

The exponential parameterization was used in a theorem concerning the asymptotic joint density function of the eigenvalues of the sample variance᎐covariance matrix ŽMuirhead, 1982, page 394.. Another parameterization of Q is given by Q s Ž I y U. Ž I q U.

y1

,

where U is a skew-symmetric matrix. This relationship is valid provided that Q does not have the eigenvalue y1. Otherwise, Q can be written as Q s LŽ I y U. Ž I q U.

y1

,

where L is a diagonal matrix in which each element on the diagonal is either 1 or y1. Arthur Cayley Ž1821᎐1895. is credited with having introduced the relationship between Q and U. Finally, the recent article by Olkin Ž1990. illustrates the strong interplay between statistics and linear algebra. The author listed several areas of statistics with a strong linear algebra component.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Anderson, T. W., and S. D. Gupta Ž1963.. ‘‘Some inequalities on characteristic roots of matrices,’’ Biometrika, 50, 522᎐524. Anderson, T. W., I. Olkin, and L. G. Underhill Ž1987.. ‘‘Generation of random orthogonal matrices.’’ SIAM J. Sci. Statist. Comput., 8, 625᎐629. Basilevsky, A. Ž1983.. Applied Matrix Algebra in the Statistical Sciences. North-Holland, New York. ŽThis book addresses topics in matrix algebra that are useful in both applied and theoretical branches of the statistical sciences. . Bellman, R. Ž1970.. Introduction to Matrix Analysis, 2nd ed. McGraw-Hill, New York. ŽAn excellent reference book on matrix algebra. The minimum᎐maximum characterization of eigenvalues is discussed in Chap. 7. Kronecker products are studied in Chap. 12. Some applications of matrices to stochastic processes and probability theory are given in Chap. 14.. Belsley, D. A., E. Kuh, and R. E. Welsch Ž1980.. Regression Diagnostics. Wiley, New York. ŽThis is a good reference for learning about multicollinearity in linear statistical models that was discussed in Section 2.4.2. Examples are provided based on actual econometric data. . Bush, K. A., and I. Olkin Ž1959.. ‘‘Extrema of quadratic forms with applications to statistics.’’ Biometrika, 46, 483᎐486. Gantmacher, F. R. Ž1959.. The Theory of Matrices, Vols. I and II. Chelsea, New York. ŽThese two volumes provide a rather more advanced study of matrix algebra than standard introductory texts. Methods to parameterize an orthogonal matrix, which were mentioned in Section 2.4.4, are discussed in Vol. I..

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

51

Golub, G. H., and C. F. Van Loan Ž1983.. Matrix Computations. Johns Hopkins University Press, Baltimore, Maryland. Good, I. J. Ž1969.. ‘‘Some applications of the singular decomposition of a matrix.’’ Technometrics, 11, 823᎐831. Graybill, F. A. Ž1961.. An Introduction to Linear Statistical Models, Vol. I. McGraw-Hill, New York. ŽThis is considered a classic textbook in experimental statistics. It is concerned with the mathematical treatment, using matrix algebra, of linear statistical models.. Graybill, F. A. Ž1983.. Matrices with Applications in Statistics, 2nd ed. Wadsworth, Belmont, California. ŽThis frequently referenced textbook contains a great number of theorems in matrix algebra, and describes many properties of matrices that are pertinent to linear model and mathematical statistics. . Healy, M. J. R. Ž1986.. Matrices for Statistics. Clarendon Press, Oxford, England. ŽThis is a short book that provides a brief coverage of some basic concepts in matrix algebra. Some applications in statistics are also mentioned.. Heiberger, R. M., P. F. Velleman, and M. A. Ypelaar Ž1983.. ‘‘Generating test data with independently controllable features for multivariate general linear forms.’’ J. Amer. Statist. Assoc., 78, 585᎐595. Henderson, H. V., F. Pukelsheim, and S. R. Searle Ž1983.. ‘‘On the history of the Kronecker product.’’ Linear and Multilinear Algebra, 14, 113᎐120. Henderson, H. V., and S. R. Searle Ž1981.. ‘‘The vec-permutation matrix, the vec operator and Kronecker products: A review.’’ Linear and Multilinear Algebra, 9, 271᎐288. Hoerl, A. E., and R. W. Kennard Ž1970.. ‘‘Ridge regression: Applications to nonorthogonal problems.’’ Technometrics, 12, 69᎐82. James, A. T. Ž1954.. ‘‘Normal multivariate analysis and the orthogonal group.’’ Ann. Math. Statist., 25, 40᎐75. Khuri, A. I. Ž1982.. ‘‘Direct products: A powerful tool for the analysis of balanced data.’’ Comm. Statist. Theory Methods, 11, 2903᎐2920. Khuri, A. I. Ž1984.. ‘‘Interval estimation of fixed effects and of functions of variance components in balanced mixed models.’’ Sankhya, Series B, 46, 10᎐28. ŽSection 5 in this article gives a procedure for the construction of exact simultaneous confidence intervals on estimable linear functions of the fixed effects in a balanced mixed model.. Khuri, A. I., and I. J. Good Ž1989.. ‘‘The parameterization of orthogonal matrices: A review mainly for statisticians.’’ South African Statist. J., 23, 231᎐250. Khuri, A. I., and R. H. Myers Ž1981.. ‘‘Design related robustness of tests in regression models.’’ Comm. Statist. Theory Methods, 10, 223᎐235. Lancaster, P. Ž1969.. Theory of Matrices. Academic Press, New York. ŽThis book is written primarily for students of applied mathematics, engineering, or science who want to acquire a good knowledge of the theory of matrices. Chap. 7 has an interesting discussion concerning the behavior of matrix eigenvalues under perturbation of the elements of the matrix.. Lowerre, J. M. Ž1982.. ‘‘An introduction to modern matrix methods and statistics.’’ Amer. Statist., 36, 113᎐115. ŽAn application of the singular-value decomposition is given in Section 2 of this article. .

52

BASIC CONCEPTS IN LINEAR ALGEBRA

Magnus, J. R., and H. Neudecker Ž1988.. Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York. ŽThis book consists of six parts. Part one deals with the basics of matrix algebra. The remaining parts are devoted to the development of matrix differential calculus and its applications to statistics and econometrics. Part four has a chapter on inequalities concerning eigenvalues that pertains to Section 2.3.10 in this chapter. . Mandel, J. Ž1982.. ‘‘Use of the singular-value decomposition in regression analysis.’’ Amer. Statist., 36, 15᎐24. Marcus, M., and H. Minc Ž1988.. Introduction to Linear Algebra. Dover, New York. ŽThis book presents an introduction to the fundamental concepts of linear algebra and matrix theory.. Marsaglia, G., and G. P. H. Styan Ž1974.. ‘‘Equalities and inequalities for ranks of matrices.’’ Linear and Multilinear Algebra, 2, 269᎐292. ŽThis is an interesting collection of results on ranks of matrices. It includes a wide variety of equalities and inequalities for ranks of products, of sums, and of partitioned matrices. . May, W. G. Ž1970.. Linear Algebra. Scott, Foresman and Company, Glenview, Illinois. Montgomery, D. C., and E. A. Peck Ž1982.. Introduction to Linear Regression Analysis. Wiley, New York. ŽChap. 8 in this book has an interesting discussion concerning multicollinearity. It includes the sources of multicollinearity, its harmful effects in regression, available diagnostics, and a survey of remedial measures. This chapter provides useful additional information to the material in Section 2.4.2.. Morrison, D. F. Ž1967.. Multi®ariate Statistical Methods. McGraw-Hill, New York. ŽThis book can serve as an introductory text to multivariate analysis. . Muirhead, R. J. Ž1982.. Aspects of Multi®ariate Statistical Theory. Wiley, New York. ŽThis book is designed as a text for a graduate-level course in multivariate analysis. . Myers, R. H. Ž1990.. Classical and Modern Regression with Applications, 2nd ed. PWS-Kent, Boston. ŽChap. 8 in this book should be useful reading concerning multicollinearity and its effects. . Newcomb, R. W. Ž1960.. ‘‘On the simultaneous diagonalization of two semidefinite matrices.’’ Quart. Appl. Math., 19, 144᎐146. Olkin, I. Ž1990.. ‘‘Interface between statistics and linear algebra.’’ In Matrix Theory and Applications, Vol. 40, C. R. Johnson, ed., American Mathematical Society, Providence, Rhode Island, pp. 233᎐256. Price, G. B. Ž1947.. ‘‘Some identities in the theory of determinants.’’ Amer. Math. Monthly, 54, 75᎐90. ŽSection 10 in this article gives some history of the theory of determinants. . Rogers, G. S. Ž1984.. ‘‘Kronecker products in ANOVAᎏa first step.’’ Amer. Statist., 38, 197᎐202. Searle, S. R. Ž1971.. Linear Models. Wiley, New York. Searle, S. R. Ž1982.. Matrix Algebra Useful for Statistics. Wiley, New York. ŽThis is a useful book introducing matrix algebra in a manner that is helpful in the statistical analysis of data and in statistics in general. Chaps. 13, 14, and 15 present applications of matrices in regression and linear models.. Seber, G. A. F. Ž1984.. Multi®ariate Obser®ations. Wiley, New York. ŽThis is a good reference on applied multivariate analysis that is suited for a graduate-level course..

EXERCISES

53

Smith, D. E. Ž1958.. History of Mathematics. Vol. I. Dover, New York. ŽThis interesting book contains, among other things, some history concerning the development of the theory of determinants and matrices. . Wolkowicz, H., and G. P. H. Styan Ž1980.. ‘‘Bounds for eigenvalues using traces.’’ Linear Algebra Appl., 29, 471᎐506.

EXERCISES In Mathematics 2.1. Show that a set of n = 1 vectors, u 1 , u 2 , . . . , u m , is always linearly dependent if m ) n. 2.2. Let W be a vector subspace of V such that W s LŽu 1 , u 2 , . . . , u n ., where the u i ’s Ž i s 1, 2, . . . , n. are linearly independent. If v is any vector in V that is not in W, then the vectors u 1 , u 2 , . . . , u n , v are linearly independent. 2.3. Prove Theorem 2.1.3. 2.4. Prove part 1 of Theorem 2.2.1. 2.5. Let T : U ™ V be a linear transformation. Show that T is one-to-one if and only if whenever u 1 , u 2 , . . . , u n are linearly independent in U, then T Žu 1 ., T Žu 2 ., . . . , T Žu n . are linearly independent in V. 2.6. Let T : R n ™ R m be represented by an n = m matrix of rank ␳ . (a) Show that dimw T Ž R n .x s ␳ . (b) Show that if n F m and ␳ s n, then T is one-to-one. 2.7. Show that trŽA⬘A. s 0 if and only if A s 0. 2.8. Let A be a symmetric positive semidefinite matrix of order n = n. Show that v⬘Av s 0 if and only if Av s 0. 2.9. The matrices A and B are symmetric and positive semidefinite of order n = n such that AB s BA. Show that AB is positive semidefinite. 2.10. If A is a symmetric n = n matrix, and B is an n = n skew-symmetric matrix, then show that trŽAB. s 0. 2.11. Suppose that trŽPA. s 0 for every skew-symmetric matrix P. Show that the matrix A is symmetric.

54

BASIC CONCEPTS IN LINEAR ALGEBRA

2.12. Let A be an n = n matrix and C be a nonsingular matrix of order n = n. Show that A, Cy1AC, and CACy1 have the same set of eigenvalues. 2.13. Let A be an n = n symmetric matrix, and let ␭ be an eigenvalue of A of multiplicity k. Then A y ␭I n has rank n y k. 2.14. Let A be a nonsingular matrix of order n = n, and let c and d be n = 1 vectors. If d⬘Ay1 c / y1, then

Ž A q cd⬘ .

y1

s Ay1 y

Ž Ay1 c . Ž d⬘Ay1 . 1 q d⬘Ay1 c

.

This is known as the Sherman-Morrison formula. 2.15. Show that if A and I k q V⬘Ay1 U are nonsingular, then

Ž A q UV⬘.

y1

s Ay1 y Ay1 U Ž I k q V⬘Ay1 U .

y1

V⬘Ay1 ,

where A is of order n = n, and U and V are of order n = k. This result is known as the Sherman-Morrison-Woodbury formula and is a generalization of the result in Exercise 2.14. 2.16. Prove Theorem 2.3.17. 2.17. Let A and B be n = n idempotent matrices. Show that A y B is idempotent if and only if AB s BA s B. 2.18. Let A be an orthogonal matrix. What can be said about the eigenvalues of A? 2.19. Let A be a symmetric matrix of order n = n, and let L be a matrix of order n = m. Show that emin Ž A . tr Ž L⬘L. F tr Ž L⬘AL. F emax Ž A . tr Ž L⬘L. 2.20. Let A be a nonnegative definite matrix of order n = n, and let L be a matrix of order n = m. Show that (a) emin ŽL⬘AL. G emin ŽA. emin ŽL⬘L., (b) emax ŽL⬘AL. F emax ŽA. emax ŽL⬘L.. 2.21. Let A and B be n = n symmetric matrices with A nonnegative definite. Show that emin Ž B . tr Ž A . F tr Ž AB . F emax Ž B . tr Ž A . .

55

EXERCISES

2.22. Let (a ) (b) (c )

Ay be a g-inverse of A. Show that Ay A is idempotent, r ŽAy. G r ŽA., r ŽA. s r ŽAy A..

In Statistics 2.23. Let y s Ž y 1 , y 2 , . . . , yn .⬘ be a normal random vector N Ž0, ␴ 2 I n .. Let y and s 2 be the sample mean and sample variance given by ys

s2 s

n

1

Ý yi ,

n

is1

1 ny1

n

Ý

yi2 y

Ž Ýnis1 yi .

is1

n

2

.

(a) Show that A is an idempotent matrix of rank n y 1, where A is an n = n matrix such that y⬘Ays Ž n y 1. s 2 . (b) What distribution does Ž n y 1. s 2r␴ 2 have? (c) Show that y and Ay are uncorrelated; then conclude that y and s 2 are statistically independent. 2.24. Consider the one-way classification model yi j s ␮ q ␣ i q ⑀ i j ,

i s 1, 2, . . . , a;

j s 1, 2, . . . , n i ,

where ␮ and ␣ i Ž i s 1, 2, . . . , a. are unknown parameters and ⑀ i j is a random error with a zero mean. Show that (a) ␣ i y ␣ i⬘ is an estimable linear function for all i / i⬘ Ž i, i⬘ s 1, 2, . . . , a., (b) ␮ is nonestimable. 2.25. Consider the linear model y s X␤ q ⑀ , where X is a known matrix of order n = p and rank r ŽF p ., ␤ is an unknown parameter vector, and ⑀ is a random error vector such that E Ž ⑀ . s 0 and VarŽ ⑀ . s ␴ 2 I n . (a) Show that XŽX⬘X.y X⬘ is an idempotent matrix. (b) Let l⬘y be an unbiased linear estimator of ␭⬘␤. Show that

ˆ . F Var Ž l⬘y. , Var Ž ␭⬘␤ ˆ s ␭⬘ŽX⬘X.y X⬘y. where ␭⬘␤ The result given in part Žb. is known as the Gauss᎐Marko® theorem.

56

BASIC CONCEPTS IN LINEAR ALGEBRA

2.26. Consider the linear model in Exercise 2.25, and suppose that r ŽX. s p. Hoerl and Kennard Ž1970. introduced an estimator of ␤ called the ridge estimator ␤*: ␤* s Ž X⬘X q kI p .

y1

X⬘y,

where k is a ‘‘small’’ fixed number. For an appropriate value of k, ␤* provides improved accuracy in the estimation of ␤ over the least-squares ˆ s ŽX⬘X.y1 X⬘y. Let X⬘X s P⌳ P⬘ be the spectral decomposiestimator ␤ ˆ where D is a diagonal matrix tion of X⬘X. Show that ␤* s PDP⬘␤, whose ith diagonal element is ␭ irŽ ␭ i q k ., i s 1, 2, . . . , p, and where ␭1 , ␭2 , . . . , ␭ p are the diagonal elements of ⌳ . 2.27. Consider the ratio

Ž x⬘Ay. , Ž x⬘B 1 x . Ž y⬘B 2 y. 2

␳2s

where A is a matrix of order m = n and B 1 , B 2 are positive definite of orders m = m and n = n, respectively. Show that y1 sup ␳ 2 s emax Ž By1 1 AB 2 A⬘ . .

x, y

w Hint: Define C 1 and C 2 as symmetric nonsingular matrices such that C 12 s B 1 , C 22 s B 2 . Let C 1 x s u, C 2 y s v. Then ␳ 2 can be written as

␳ s 2

y1 Ž u⬘Cy1 1 AC 2 v .

Ž u⬘u. Ž v⬘v .

2 y1 s Ž ␯⬘Cy1 1 AC 2 ␶ . , 2

where ␯ s urŽu⬘u.1r2 , ␶ s vrŽv⬘v.1r2 are unit vectors. Verify the result of this problem after noting that ␳ 2 is now the square of a dot product. x Note: This exercise has the following application in multivariate analysis: Let z 1 and z 2 be random vectors with zero means and variance᎐ covariance matrices ⌺ 11 , ⌺ 22 , respectively. Let ⌺ 12 be the covariance matrix of z 1 and z 2 . On choosing A s ⌺ 12 , B 1 s ⌺ 11 , B 2 s ⌺ 22 , the positive square root of the supremum of ␳ 2 is called the canonical correlation coefficient between z 1 and z 2 . It is a measure of the linear association between z 1 and z 2 Žsee, for example, Seber, 1984, Section 5.7..

CHAPTER 3

Limits and Continuity of Functions

The notions of limits and continuity of functions lie at the kernel of calculus. The general concept of continuity is very old in mathematics. It had its inception long ago in ancient Greece. We owe to Aristotle Ž384᎐322 B.C.. the first known definition of continuity: ‘‘A thing is continuous when of any two successive parts the limits at which they touch are one and the same and are, as the word implies, held together’’ Žsee Smith, 1958, page 93.. Our present definitions of limits and continuity of functions, however, are substantially those given by Augustin-Louis Cauchy Ž1789᎐1857.. In this chapter we introduce the concepts of limits and continuity of real-valued functions, and study some of their properties. The domains of definition of the functions will be subsets of R, the set of real numbers. A typical subset of R will be denoted by D.

3.1. LIMITS OF A FUNCTION Before defining the notion of a limit of a function, let us understand what is meant by the notation x™ a, where a and x are elements in R. If a is finite, then x™ a means that x can have values that belong to a neighborhood Nr Ž a. of a Žsee Definition 1.6.1. for any r ) 0, but x / a, that is, 0 - < xy a < r. Such a neighborhood is called a deleted neighborhood of a, that is, a neighborhood from which the point a has been removed. If a is infinite Žy⬁ or q⬁., then x™ a indicates that < x < can get larger and larger without any constraint on the extent of its increase. Thus < x < can have values greater than any positive number. In either case, whether a is finite or infinite, we say that x tends to a or approaches a. Let us now study the behavior of a function f Ž x . as x ™ a. Definition 3.1.1. Suppose that the function f Ž x . is defined in a deleted neighborhood of a point ag R. Then f Ž x . is said to have a limit L as x™ a 57

58

LIMITS AND CONTINUITY OF FUNCTIONS

if for every ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x. yL -⑀

Ž 3.1 .

0 - < xy a < - ␦ .

Ž 3.2 .

for all x for which

In this case, we write f Ž x . ™ L as x™ a, which is equivalent to saying that lim x ™ a f Ž x . s L. Less formally, we say that f Ž x . ™ L as x™ a if, however small the positive number ⑀ might be, f Ž x . differs from L by less than ⑀ for values of x sufficiently close to a. I NOTE 3.1.1. When f Ž x . has a limit as x™ a, it is considered to be finite. If this is not the case, then f Ž x . is said to have an infinite limit Žy⬁ or q⬁. as x™ a. This limit exists only in the extended real number system, which consists of the real number system combined with the two symbols, y⬁ and q⬁. In this case, for every positive number M there exists a ␦ ) 0 such that < f Ž x .< ) M if 0 - < xy a < - ␦ . If a is infinite and L is finite, then f Ž x . ™ L as x ™ a if for any ⑀ ) 0 there exists a positive number N such that inequality Ž3.1. is satisfied for all x for which < x < ) N. In case both a and L are infinite, then f Ž x . ™ L as x™ a if for any B ) 0 there exists a positive number A such that < f Ž x .< ) B if < x < ) A. NOTE 3.1.2. If f Ž x . has a limit L as x™ a, then L must be unique. To show this, suppose that L1 and L2 are two limits of f Ž x . as x™ a. Then, for any ⑀ ) 0 there exist ␦ 1 ) 0, ␦ 2 ) 0 such that f Ž x . y L1 f Ž x . y L2 -

⑀ 2 ⑀ 2

,

if 0 - < xy a < - ␦ 1 ,

,

if 0 - < x y a < - ␦ 2 .

Hence, if ␦ s minŽ ␦ 1 , ␦ 2 ., then < L1 y L 2 < s L1 y f Ž x . q f Ž x . y L 2 F f Ž x . y L1 q f Ž x . y L 2 -⑀ for all x for which 0 - < xy a < - ␦ . Since < L1 y L2 < is smaller than ⑀ , which is an arbitrary positive number, we must have L1 s L2 Žwhy?.. NOTE 3.1.3. The limit of f Ž x . as described in Definition 3.1.1 is actually called a two-sided limit. This is because x can approach a from either side. There are, however, cases where f Ž x . can have a limit only when x approaches a from one side. Such a limit is called a one-sided limit.

59

LIMITS OF A FUNCTION

By definition, if f Ž x . has a limit as x approaches a from the left, symbolically written as x™ ay, then f Ž x . has a left-sided limit, which we denote by Ly. In this case we write lim f Ž x . s Ly.

x™a y

If, however, f Ž x . has a limit as x approaches a from the right, symbolically written as x™ aq, then f Ž x . has a right-sided limit, denoted by Lq, that is, lim f Ž x . s Lq.

x™a q

From the above definition it follows that f Ž x . has a left-sided limit Ly as x™ ay if for every ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x . y Ly - ⑀ for all x for which 0 - ay x- ␦ . Similarly, f Ž x . has a right-sided limit Lq as x ™ aq if for every ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x . y Lq - ⑀ for all x for which 0 - xy a- ␦ . Obviously, if f Ž x . has a two-sided limit L as x ™ a, then Ly and Lq both exist and are equal to L. Vice versa, if Lys Lq, then f Ž x . has a two-sided limit L as x™ a, where L is the common value of Ly and Lq Žwhy?.. We can then state that lim x ™ a f Ž x . s L if and only if lim f Ž x . s limq f Ž x . s L.

x™a y

x™a

Thus to determine if f Ž x . has a limit as x™ a, we first need to find out if it has a left-sided limit Ly and a right-sided limit Lq as x™ a. If this is the case and Lys Lqs L, then f Ž x . has a limit L as x ™ a. Throughout the remainder of the book, we shall drop the characterization ‘‘two-sided’’ when making a reference to a two-sided limit L of f Ž x .. Instead, we shall simply state that L is the limit of f Ž x .. EXAMPLE 3.1.1.

Consider the function

f Ž x. s

½

Ž xy 1 . r Ž x 2 y 1 . , 4,

x/ y1, 1, xs 1.

This function is defined everywhere except at xs y1. Let us find its limit as x ™ a, where ag R. We note that lim f Ž x . s lim

x™a

x™a

xy 1 x y1 2

s lim

x™a

1 xq 1

.

60

LIMITS AND CONTINUITY OF FUNCTIONS

This is true even if as 1, because x/ a as x™ a. We now claim that if a/ y1, then lim

x™a

1 xq 1

s

1 aq 1

.

To prove this claim, we need to find a ␦ ) 0 such that for any ⑀ ) 0, 1 xq 1

y

1 aq 1

Ž 3.3 .

-⑀

if 0 - < xy a < - ␦ . Let us therefore consider the following two cases: CASE 1.

a) y1. In this case we have 1 xq 1

y

1

s

aq 1

< xy a < < xq 1 < < aq 1 <

Ž 3.4 .

.

If < xy a < - ␦ , then

Ž 3.5 .

ay ␦ q 1 - xq 1 - aq ␦ q 1.

Since aq 1 ) 0, we can choose ␦ ) 0 such that ay ␦ q 1 ) 0, that is, ␦ - aq 1. From Ž3.4. and Ž3.5. we then get 1 xq 1

y

1 aq 1

-

␦

Ž aq 1 . Ž ay ␦ q 1 .

.

Let us constrain ␦ even further by requiring that

␦

Ž aq 1 . Ž ay ␦ q 1 .

-⑀ .

This is accomplished by choosing ␦ ) 0 so that

Ž aq 1 . ⑀ ␦. 1 q Ž aq 1 . ⑀ 2

Since

Ž aq 1 . ⑀ - aq 1, 1 q Ž aq 1 . ⑀ 2

inequality Ž3.3. will be satisfied by all x for which < xy a < - ␦ , where

Ž aq 1 . ⑀ 0-␦. 1 q Ž aq 1 . ⑀ 2

Ž 3.6 .

61

LIMITS OF A FUNCTION

CASE 2. a- y1. Here, we choose ␦ ) 0 such that aq ␦ q 1 - 0, that is, ␦ - yŽ aq 1.. From Ž3.5. we conclude that < x q 1 < ) y Ž aq ␦ q 1 . . Hence, from Ž3.4. we get 1 xq 1

y

1 aq 1

-

␦

Ž aq 1 . Ž aq ␦ q 1 .

.

As before, we further constrain ␦ by requiring that it satisfy the inequality

␦

-⑀ ,

Ž aq 1 . Ž aq ␦ q 1 . or equivalently, the inequality

Ž aq 1 . ⑀ ␦. 1 y Ž aq 1 . ⑀ 2

Note that

Ž aq 1 . ⑀ - y Ž aq 1 . . 1 y Ž aq 1 . ⑀ 2

Consequently, inequality Ž3.3. can be satisfied by choosing ␦ such that

Ž aq 1 . ⑀ 0-␦. 1 y Ž aq 1 . ⑀ 2

Ž 3.7 .

Cases 1 and 2 can be combined by rewriting Ž3.6. and Ž3.7. using the single double inequality 0-␦-

< aq 1 < 2⑀ 1 q < aq 1 < ⑀

.

If as y1, then no limit exists as x™ a. This is because lim f Ž x . s lim

x™y1

x™y1

1 xq 1

.

62

LIMITS AND CONTINUITY OF FUNCTIONS

If x™ y1y, then lim y

x™y1

1 xq 1

s y⬁,

and, as x™ y1q, lim q

x™y1

1 xq 1

s ⬁.

Since the left-sided and right-sided limits are not equal, no limit exists as x™ y1. EXAMPLE 3.1.2.

Let f Ž x . be defined as f Ž x. s

½

1 q 'x , x,

xG 0, x- 0.

This function has no limit as x™ 0, since lim f Ž x . s limy xs 0,

x™0 y

x™0

lim f Ž x . s limq Ž 1 q 'x . s 1.

x™0 q

x™0

However, for any a/ 0, lim x ™ a f Ž x . exists. EXAMPLE 3.1.3.

Let f Ž x . be given by f Ž x. s

½

Figure 3.1. The graph of the function f Ž x ..

x cos x, 0,

x/ 0, xs 0.

SOME PROPERTIES ASSOCIATED WITH LIMITS OF FUNCTIONS

63

Then lim x ™ 0 f Ž x . s 0. This is true because < f Ž x .< F < x < in any deleted neighborhood of as 0. As x™ ⬁, f Ž x . oscillates unboundedly, since yxF x cos xF x. Thus f Ž x . has no limit as x™ ⬁. A similar conclusion can be reached when x ™ y⬁ Žsee Figure 3.1..

3.2. SOME PROPERTIES ASSOCIATED WITH LIMITS OF FUNCTIONS The following theorems give some fundamental properties associated with function limits. Theorem 3.2.1. Let f Ž x . and g Ž x . be real-valued functions defined on D ; R. Suppose that lim x ™ a f Ž x . s L and lim x ™ a g Ž x . s M. Then 1. 2. 3. 4.

lim x ™ aw f Ž x . q g Ž x .x s L q M, lim x ™ aw f Ž x . g Ž x .x s LM, lim x ™ aw1rg Ž x .x s 1rM if M/ 0, lim x ™ aw f Ž x .rg Ž x .x s LrM if M/ 0.

Proof. We shall only prove parts 2 and 3. The proof of part 1 is straightforward, and part 4 results from applying parts 2 and 3. Proof of Part 2. Consider the following three cases: CASE 1. such that

Both L and M are finite. Let ⑀ ) 0 be given and let ␶ ) 0 be

␶ Ž␶q < L< q < M< . -⑀ .

Ž 3.8 .

This inequality is satisfied by all values of ␶ for which

0-␶-

yŽ < L < q < M < . q

'Ž < L < q < M < . q 4⑀ . 2

2

Now, there exist ␦ 1 ) ␦ 2 ) 0 such that f Ž x. yL -␶

if 0 - < xy a < - ␦ 1 ,

g Ž x. yM -␶

if 0 - < xy a < - ␦ 2 .

64

LIMITS AND CONTINUITY OF FUNCTIONS

Then, for any x such that 0 - < xy a < - ␦ where ␦ s minŽ ␦ 1 , ␦ 2 ., f Ž x . g Ž x . y LM s M f Ž x . y L q f Ž x . g Ž x . y M F ␶ < M < q␶ f Ž x . F ␶ < M < q␶ < L < q f Ž x . y L F␶ Ž␶q < L< q < M< . -⑀ , which proves part 2. CASE 2. One of L and M is finite and the other is infinite. Without any loss of generality, we assume that L is finite and Ms ⬁. Let us also assume that L / 0, since 0 ⭈ ⬁ is indeterminate. Let A ) 0 be given. There exists a ␦ 1 ) 0 such that < f Ž x .< ) < L < r2 if 0 - < xy a < - ␦ 1 Žwhy?.. Also, there exists a ␦ 2 ) 0 such that < g Ž x .< ) 2 Ar < L < if 0 - < xy a < - ␦ 2 . Let ␦ s minŽ ␦ 1 , ␦ 2 .. If 0 - < xy a < - ␦ , then f Ž x. gŽ x. s f Ž x. gŽ x. -

< L<

2

2

< L<

A s A.

This means that lim x ™ a f Ž x . g Ž x . s ⬁, which proves part 2. CASE 3. Both L and M are infinite. Suppose that L s ⬁, Ms ⬁. In this case, for a given B ) 0 there exist ␬ 1 ) 0, ␬ 2 ) 0 such that f Ž x . ) 'B

if 0 - < xy a < - ␬ 1 ,

g Ž x . ) 'B

if 0 - < xy a < - ␬ 2 .

Then, f Ž x . g Ž x . ) B,

if 0 - < xy a < - ␬ ,

where ␬ s minŽ ␬ 1 , ␬ 2 .. This implies that lim x ™ a f Ž x . g Ž x . s ⬁, which proves part 2. Proof of Part 3. Let ⑀ ) 0 be given. If M/ 0, then there exists a ␭1 ) 0 such that < g Ž x .< ) < M < r2 if 0 - < xy a < - ␭1. Also, there exists a ␭2 ) 0 such that < g Ž x . y M < - ⑀ M 2r2 if 0 - < xy a < - ␭2 . Then, 1 gŽ x.

y

1 M

s

-

g Ž x. yM g Ž x. < M< 2 g Ž x. yM < M< 2

-⑀ , if 0 - < xy a < - ␭, where ␭ s minŽ ␭1 , ␭2 ..

I

THE

O

,

65

O NOTATION

Theorem 3.2.1 is also true if L and M are one-sided limits of f Ž x . and g Ž x ., respectively. If f Ž x . F g Ž x ., then lim x ™ a f Ž x . F lim x ™ a g Ž x ..

Theorem 3.2.2.

Proof. Let lim x ™ a f Ž x . s L, lim x ™ a g Ž x . s M. Suppose that L y M) 0. By Theorem 3.2.1, L y M is the limit of the function hŽ x . s f Ž x . y g Ž x .. Therefore, there exists a ␦ ) 0 such that hŽ x . y Ž L y M . -

LyM 2

,

Ž 3.9 .

if 0 - < x y a < - ␦ . Inequality Ž3.9. implies that hŽ x . ) Ž L y M .r2 ) 0, which is not possible, since, by assumption, hŽ x . s f Ž x . y g Ž x . F 0. We must then have L y MF 0. I

3.3. THE o, O NOTATION These symbols provide a convenient way to describe the limiting behavior of a function f Ž x . as x tends to a certain limit. Let f Ž x . and g Ž x . be two functions defined on D ; R. The function g Ž x . is positive and usually has a simple form such as 1, x, or 1rx. Suppose that there exists a positive number K such that f Ž x . F Kg Ž x . for all xg E, where E ; D. Then, f Ž x . is said to be of an order of magnitude not exceeding that of g Ž x .. This fact is denoted by writing f Ž x. sOŽ g Ž x. . for all xg E. In particular, if g Ž x . s 1, then f Ž x . is necessarily a bounded function on E. For example, cos xs O Ž 1 .

for all x,

xs O Ž x 2 .

for large values of x,

x q 2 xs O Ž x .

for large values of x,

2

2

sin xs O Ž < x < .

for all x.

The last relationship is true because sin x x

F1

for all values of x, where x is measured in radians.

66

LIMITS AND CONTINUITY OF FUNCTIONS

Let us now suppose that the relationship between f Ž x . and g Ž x . is such that lim

x™a

f Ž x. gŽ x.

s 0.

Then we say that f Ž x . is of a smaller order of magnitude than g Ž x . in a deleted neighborhood of a. This fact is denoted by writing f Ž x . s oŽ g Ž x . .

as x™ a,

which is equivalent to saying that f Ž x . tends to zero more rapidly than g Ž x . as x™ a. The o symbol can also be used when x tends to infinity. In this case we write f Ž x . s oŽ g Ž x . .

for x) A,

where A is some positive number. For example, x 2 s oŽ x . tan x 3 s o Ž x 2 .

'x s o Ž x .

as x™ 0, as x™ 0, as x™ ⬁.

If f Ž x . and g Ž x . are any two functions such that f Ž x. gŽ x.

™1

as x™ a,

then f Ž x . and g Ž x . are said to be asymptotically equal, written symbolically f Ž x . ; g Ž x ., as x™ a. For example, x 2 ; x 2 q 3 xq 1 sin x; x

as x™ ⬁, as x™ 0.

On the basis of the above definitions, the following properties can be deduced: 1. 2. 3. 4.

O Ž f Ž x . q g Ž x .. s O Ž f Ž x .. q O Ž g Ž x ... O Ž f Ž x . g Ž x .. s O Ž f Ž x .. O Ž g Ž x ... oŽ f Ž x . g Ž x .. s O Ž f Ž x .. oŽ g Ž x ... If f Ž x . ; g Ž x . as x™ a, then f Ž x . s g Ž x . q oŽ g Ž x .. as x™ a.

3.4. CONTINUOUS FUNCTIONS A function f Ž x . may have a limit L as x™ a. This limit, however, may not be equal to the value of the function at xs a. In fact, the function may not even

67

CONTINUOUS FUNCTIONS

be defined at this point. If f Ž x . is defined at xs a and L s f Ž a., then f Ž x . is said to be continuous at x s a. Definition 3.4.1. Let f : D ™ R, where D ; R, and let ag D. Then f Ž x . is continuous at xs a if for every ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x . y f Ž a. - ⑀ for all xg D for which < xy a < - ␦ . It is important here to note that in order for f Ž x . to be continuous at x s a, it is necessary that it be defined at xs a as well as at all other points inside a neighborhood Nr Ž a. of the point a for some r ) 0. Thus to show continuity of f Ž x . at xs a, the following conditions must be verified: 1. f Ž x . is defined at all points inside a neighborhood of the point a. 2. f Ž x . has a limit from the left and a limit from the right as x ™ a, and that these two limits are equal to L. 3. The value of f Ž x . at xs a is equal to L. For convenience, we shall denote the left-sided and right-sided limits of f Ž x . as x™ a by f Ž ay. and f Ž aq. , respectively. If any of the above conditions is violated, then f Ž x . is said to be discontinuous at xs a. There are two kinds of discontinuity. I Definition 3.4.2. A function f : D ™ R has a discontinuity of the first kind at xs a if f Ž ay. and f Ž aq. exist, but at least one of them is different from f Ž a.. The function f Ž x . has a discontinuity of the second kind at the same point if at least one of f Ž ay. and f Ž aq. does not exist. I Definition 3.4.3. A function f : D ™ R is continuous on E ; D if it is continuous at every point of E. For example, the function

° xy 1 ,

~x

f Ž x. s

2

y1

¢2 , 1

xG 0, x / 1, xs 1,

is defined for all xG 0 and is continuous at xs 1. This is true because, as was shown in Example 3.1.1, lim f Ž x . s lim

x™1

x™1

1 xq 1

s

1 2

,

which is equal to the value of the function at xs 1. Furthermore, f Ž x . is

68

LIMITS AND CONTINUITY OF FUNCTIONS

continuous at all other points of its domain. Note that if f Ž1. were different from 12 , then f Ž x . would have a discontinuity of the first kind at x s 1. Let us now consider the function

½

xq 1, f Ž x . s 0, xy 1,

x) 0, xs 0, x- 0.

This function is continuous everywhere except at xs 0, since it has no limit as x™ 0 by the fact that f Ž0y. s y1 and f Ž0q. s 1. The discontinuity at this point is therefore of the first kind. An example of a discontinuity of the second kind is given by the function

° ¢0,

1

cos , f Ž x . s~ x

x/ 0, xs 0,

which has a discontinuity of the second kind at x s 0, since neither f Ž0y. nor f Ž0q. exists. Definition 3.4.4. A function f : D ™ R is left-continuous at x s a if lim x ™ ay f Ž x . s f Ž a.. It is right-continuous at xs a if lim x ™ aq f Ž x . s f Ž a.. I Obviously, a left-continuous or a right-continuous function is not necessarily continuous. In order for f Ž x . to be continuous at xs a it is necessary and sufficient that f Ž x . be both left-continuous and right-continuous at this point. For example, the function f Ž x. s

½

xy 1, 1,

xF 0, x) 0

is left-continuous at xs 0, since f Ž0y. s y1 s f Ž0.. If f Ž x . were defined so that f Ž x . s xy 1 for x- 0 and f Ž x . s 1 for xG 0, then it would be right-continuous at xs 0. Definition 3.4.5. The function f : D ™ R is uniformly continuous on E ; D if for every ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x1 . y f Ž x 2 . - ⑀ for all x 1 , x 2 g E for which < x 1 y x 2 < - ␦ .

Ž 3.10 . I

This definition appears to be identical to the definition of continuity. That is not exactly the case. Uniform continuity is always associated with a set such

69

CONTINUOUS FUNCTIONS

as E in Definition 3.4.5, whereas continuity can be defined at a single point a. Furthermore, inequality Ž3.10. is true for all pairs of points x 1 , x 2 g E such that < x 1 y x 2 < - ␦ . Hence, ␦ depends only on ⑀ , not on the particular locations of x 1 , x 2 . On the other hand, in the definition of continuity ŽDefinition 3.4.1. ␦ depends on ⑀ as well as on the location of the point where continuity is considered. In other words, ␦ can change from one point to another for the same given ⑀ ) 0. If, however, for a given ⑀ ) 0, the same ␦ can be used with all points in some set E ; D, then f Ž x . is uniformly continuous on E. For this reason, whenever f Ž x . is uniformly continuous on E, ␦ can be described as being ‘‘portable,’’ which means it can be used everywhere inside E provided that ⑀ ) 0 remains unchanged. Obviously, if f Ž x . is uniformly continuous on E, then it is continuous there. The converse, however, is not true. For example, consider the function f : Ž0, 1. ™ R given by f Ž x . s 1rx. Here, f Ž x . is continuous at all points of E s Ž0, 1., but is not uniformly continuous there. To demonstrate this fact, let us first show that f Ž x . is continuous on E. Let ⑀ ) 0 be given and let ag E. Since a) 0, there exists a ␦ 1 ) 0 such that the neighborhood N␦ 1Ž a. is a subset of E. This can be accomplished by choosing ␦ 1 such that 0 - ␦ 1 - a. Now, for all xg N␦ 1Ž a., 1 x

y

1 a

s

< xy a <

-

ax

␦1 a Ž ay ␦ 1 .

.

Let ␦ 2 ) 0 be such that for the given ⑀ ) 0,

␦2 a Ž ay ␦ 2 .

-⑀ ,

which can be satisfied by requiring that 0 - ␦2 -

a2⑀ 1 q a⑀

.

Since a2⑀ 1 q a⑀

- a,

then 1 x

y

1 a

-⑀

Ž 3.11 .

70

LIMITS AND CONTINUITY OF FUNCTIONS

if < xy a < - ␦ , where

ž

␦ - min a,

a2⑀ 1 q a⑀

/

s

a2⑀ 1 q a⑀

.

It follows that f Ž x . s 1rx is continuous at every point of E. We note here the dependence of ␦ on both ⑀ and a. Let us now demonstrate that f Ž x . s 1rx is not uniformly continuous on E. Define G to be the set Gs

½

a2⑀ 1 q a⑀

5

ag E .

In order for f Ž x . s 1rx to be uniformly continuous on E, the infimum of G must be positive. If this is possible, then for a given ⑀ ) 0, Ž3.11. will be satisfied by all x for which < xy a < - inf Ž G . , and for all ag Ž0, 1.. However, this cannot happen, since infŽ G . s 0. Thus it is not possible to find a single ␦ for which Ž3.11. will work for all ag Ž0, 1.. Let us now try another function defined on the same set E s Ž0, 1., namely, the function f Ž x . s x 2 . In this case, for a given ⑀ ) 0, let ␦ ) 0 be such that

␦ 2 q 2 ␦ ay ⑀ - 0.

Ž 3.12 .

Then, for any ag E, if < xy a < - ␦ we get < x 2 y a2 < s < xy a < < x q a < s < xy a < < x y aq 2 a < F ␦ Ž ␦ q 2 a. - ⑀ .

Ž 3.13 .

It is easy to see that this inequality is satisfied by all ␦ ) 0 for which 0 - ␦ - yaq 'a2 q ⑀ .

Ž 3.14 .

If H is the set H s
yaq 'a2 q ⑀ < ag E 4 , then it can be verified that infŽ H . s y1 q '1 q ⑀ . Hence, by choosing ␦ such that

␦ F inf Ž H . , inequality Ž3.14., and hence Ž3.13., will be satisfied for all ag E. The function f Ž x . s x 2 is therefore uniformly continuous on E.

71

CONTINUOUS FUNCTIONS

The above examples demonstrate that continuity and uniform continuity on a set E are not always equivalent. They are, however, equivalent under certain conditions on E. This will be illustrated in Theorem 3.4.6 in the next subsection. 3.4.1. Some Properties of Continuous Functions Continuous functions have some interesting properties, some of which are given in the following theorems: Theorem 3.4.1. Let f Ž x . and g Ž x . be two continuous functions defined on a set D ; R. Then: 1. f Ž x . q g Ž x . and f Ž x . g Ž x . are continuous on D. 2. af Ž x . is continuous on D, where a is a constant. 3. f Ž x .rg Ž x . is continuous on D provided that g Ž x . / 0 on D. Proof. The proof is left as an exercise.

I

Theorem 3.4.2. Suppose that f : D ™ R is continuous on D, and g: f Ž D . ™ R is continuous on f Ž D ., the image of D under f. Then the composite function h: D ™ R defined as hŽ x . s g w f Ž x .x is continuous on D. Proof. Let ⑀ ) 0 be given, and let ag D. Since g is continuous at f Ž a., there exists a ␦ ⬘ ) 0 such that < g w f Ž x .x y g w f Ž a.x< - ⑀ if < f Ž x . y f Ž a.< - ␦ ⬘. Since f Ž x . is continuous at xs a, there exists a ␦ ) 0 such that < f Ž x . y f Ž a.< - ␦ ⬘ if < xy a < - ␦ . It follows that by taking < xy a < - ␦ we must have < hŽ x . y hŽ a.< - ⑀ . I Theorem 3.4.3. If f Ž x . is continuous at xs a and f Ž a. ) 0, then there exists a neighborhood N␦ Ž a. in which f Ž x . ) 0. Proof. Since f Ž x . is continuous at xs a, there exists a ␦ ) 0 such that f Ž x . y f Ž a . - 12 f Ž a . , if < xy a < - ␦ . This implies that f Ž x . ) 12 f Ž a . ) 0 for all xg N␦ Ž a..

I

Theorem 3.4.4 ŽThe Intermediate-Value Theorem.. Let f : D ™ R be continuous, and let w a, b x be a closed interval contained in D. Suppose that

72

LIMITS AND CONTINUITY OF FUNCTIONS

f Ž a. - f Ž b .. If ␭ is a number such that f Ž a. - ␭ - f Ž b ., then there exists a point c, where a- c - b, such that ␭ s f Ž c .. Proof. Let g: D ™ R be defined as g Ž x . s f Ž x . y ␭. This function is continuous and is such that g Ž a. - 0, g Ž b . ) 0. Consider the set S s
xg w a, b x < g Ž x . - 0 4 . This set is nonempty, since ag S and is bounded from above by b. Hence, by Theorem 1.5.1 the least upper bound of S exists. Let c s lubŽ S .. Since S ; w a, b x, then c g w a, b x. Now, for every positive integer n, there exists a point x n g S such that cy

1 n

- x n F c.

Otherwise, if xF c y 1rn for all xg S, then c y 1rn will be an upper bound of S, contrary to the definition of c. Consequently, lim n™⬁ x n s c. Since g Ž x . is continuous on w a, b x, then g Ž c . s lim g Ž x n . F 0, x n™c

Ž 3.15 .

by Theorem 3.2.2 and the fact that g Ž x n . - 0. From Ž3.15. we conclude that c - b, since g Ž b . ) 0. Let us suppose that g Ž c . - 0. Then, by Theorem 3.4.3, there exists a neighborhood N␦ Ž c ., for some ␦ ) 0, such that g Ž x . - 0 for all x g N␦ Ž c . l w a, b x. Consequently, there exists a point x 0 g w a, b x such that c - x 0 - c q ␦ and g Ž x 0 . - 0. This means that x 0 belongs to S and is greater than c, a contradiction. Therefore, by inequality Ž3.15. we must have g Ž c . s 0, that is, f Ž c . s ␭. We note that c ) a, since c G a, but c / a. This last is true because if as c, then g Ž c . - 0, a contradiction. This completes the proof of the theorem. I The direct implication of the intermediate-value theorem is that a continuous function possesses the property of assuming at least once every value between any two distinct values taken inside its domain. Theorem 3.4.5. Suppose that f : D ™ R is continuous and that D is closed and bounded. Then f Ž x . is bounded in D. Proof. Let a be the greatest lower bound of D, which exists because D is bounded. Since D is closed, then ag D Žwhy?.. Furthermore, since f Ž x .

73

CONTINUOUS FUNCTIONS

is continuous, then for a given ⑀ ) 0 there exists a ␦ 1 ) 0 such that f Ž a. y ⑀ - f Ž x . - f Ž a. q ⑀ if < xy a < - ␦ 1. The function f Ž x . is therefore bounded in N␦ 1Ž a.. Define A to be the set A s
xg D < f Ž x . is bounded 4 . This set is nonempty and bounded, and N␦ 1Ž a. l D ; A. We need to show that D y A is an empty set. As before, the least upper bound of A exists Žsince it is bounded. and belongs to D Žsince D is closed.. Let c s lubŽ A .. By the continuity of f Ž x ., there exists a neighborhood N␦ 2Ž c . in which f Ž x . is bounded for some ␦ 2 ) 0. If D y A is nonempty, then N␦ 2Ž c . l Ž D y A . is also nonempty wif N␦ 2Ž c . ; A, then c q Ž ␦ 2r2. g A, a contradiction x. Let x 0 g N␦ 2Ž c . l Ž D y A .. Then, on one hand, f Ž x 0 . is not bounded, since x 0 g Ž D y A .. On the other hand, f Ž x 0 . must be bounded, since x 0 g N␦ 2Ž c .. This contradiction leads us to conclude that D y A must be empty and that f Ž x . is bounded in D. I Corollary 3.4.1. If f : D ™ R is continuous, where D is closed and bounded, then f Ž x . achieves its infimum and supremum at least once in D, that is, there exists ␰ , ␩ g D such that f Ž ␰ . Ff Ž x.

for all xg D,

f Ž␩ . Gf Ž x.

for all xg D.

Equivalently, f Ž ␰ . s inf f Ž x . , xgD

f Ž ␩ . s sup f Ž x . . xgD

Proof. By Theorem 3.4.5, f Ž D . is a bounded set. Hence, its least upper bound exists. Let Ms lub f Ž D ., which is the same as sup x g D f Ž x .. If there exists no point x in D for which f Ž x . s M, then My f Ž x . ) 0 for all xg D. Consequently, 1rw My f Ž x .x is continuous on D by Theorem 3.4.1, and is hence bounded there by Theorem 3.4.5. Now, if ␦ ) 0 is any given positive number, we can find a value x for which f Ž x . ) My ␦ , or 1 My f Ž x .

)

1

␦

.

This implies that 1rw My f Ž x .x is not bounded, a contradiction. Therefore, there must exist a point ␩ g D at which f Ž␩ . s M.

74

LIMITS AND CONTINUITY OF FUNCTIONS

The proof concerning the existence of a point ␰ g D such that f Ž ␰ . s inf x g D f Ž x . is similar. I The requirement that D be closed in Corollary 3.4.1 is essential. For example, the function f Ž x . s 2 xy 1, which is defined on D s
x < 0 - x- 14 , cannot achieve its infimum, namely y1, in D. For if there exists a ␰ g D such that f Ž ␰ . F 2 xy 1 for all xg D, then there exists a ␦ ) 0 such that 0 - ␰ y ␦ . Hence, f Ž ␰y␦ . s2␰y2␦y1-f Ž ␰ . , a contradiction. Theorem 3.4.6. Let f : D ™ R be continuous on D. If D is closed and bounded, then f is uniformly continuous on D. Proof. Suppose that f is not uniformly continuous. Then, by using the logical negation of the statement concerning uniform continuity in Definition 3.4.5, we may conclude that there exists an ⑀ ) 0 such that for every ␦ ) 0, we can find x 1 , x 2 g D with < x 1 y x 2 < - ␦ for which < f Ž x 1 . y f Ž x 2 .< G ⑀ . On this basis, by choosing ␦ s 1, we can find u1 , ®1 g D with < u1 y ®1 < - 1 for which < f Ž u1 . y f Ž ®1 .< G ⑀ . Similarly, we can find u 2 , ®2 g D with < u 2 y ®2 < - 12 for which < f Ž u 2 . y f Ž ®2 .< G ⑀ . By continuing in this process we can find u n , ®n g D with < u n y ®n < - 1rn for which < f Ž u n . y f Ž ®n .< G ⑀ , n s 3, 4, . . . . Now, let S be the set S s
u n < n s 1, 2, . . . 4 This set is bounded, since S ; D. Hence, its least upper bound exists. Let c s lubŽ S .. Since D is closed, then c g D. Thus, as in the proof of Theorem 3.4.4, we can find points u n1, u n 2 , . . . , u n k, . . . in S such that lim k ™⬁ u n k s c. Since f Ž x . is continuous, there exists a ␦ ⬘ ) 0 such that f Ž x . yf Ž c. -

⑀ 2

,

if < xy c < - ␦ ⬘ for any given ⑀ ) 0. Let us next choose k large enough such that if n k ) N, where N is some large positive number, then unk y c -

␦⬘ 2

and

1 nk

-

␦⬘ 2

.

Ž 3.16 .

Since < u n k y ®n k < - 1rn k , then ®n k y c F u n k y ®n k q u n k y c -

1 nk

q

␦⬘ 2

-␦ ⬘

Ž 3.17 .

75

CONTINUOUS FUNCTIONS

for n k ) N. From Ž3.16. and Ž3.17. and the continuity of f Ž x . we conclude that f Ž unk . y f Ž c . -

⑀ 2

f Ž ®n k . y f Ž c . -

and

⑀ 2

.

Thus, f Ž u n k . y f Ž ®n k . F f Ž u n k . y f Ž c . q f Ž ®n k . y f Ž c .

Ž 3.18 .

-⑀ . However, as was seen earlier, f Ž u n . y f Ž ®n . G ⑀ , hence, f Ž u n k . y f Ž ®n k . G ⑀ ,

which contradicts Ž3.18.. This leads us to assert that f Ž x . is uniformly continuous on D. I 3.4.2. Lipschitz Continuous Functions Lipschitz continuity is a specialized form of uniform continuity. Definition 3.4.6. The function f : D ™ R is said to satisfy the Lipschitz condition on a set E ; D if there exist constants, K and ␣ , where K ) 0 and 0 - ␣ F 1 such that f Ž x1 . y f Ž x 2 . F K < x1 y x 2 < ␣ for all x 1 , x 2 g E.

I

Notationally, whenever f Ž x . satisfies the Lipschitz condition with constants K and ␣ on a set E, we say that it is LipŽ K, ␣ . on E. In this case, f Ž x . is called a Lipschitz continuous function. It is easy to see that a Lipschitz continuous function on E is also uniformly continuous there. As an example of a Lipschitz continuous function, consider f Ž x . s 'x , where xG 0. We claim that 'x is LipŽ1, 12 . on its domain. To show this, we first write

'x

1

'

'

'

y x 2 F x1 q x 2 .

76

LIMITS AND CONTINUITY OF FUNCTIONS

Hence,

'x

1

'

y x2

2

F < x1 y x 2 < .

Thus,

'x

1

'

y x 2 F < x 1 y x 2 < 1r2 ,

which proves our claim.

3.5. INVERSE FUNCTIONS From Chapter 1 we recall that one of the basic characteristics of a function y s f Ž x . is that two values of y are equal if they correspond to the same value of x. If we were to reverse the roles of x and y so that two values of x are equal whenever they correspond to the same value of y, then x becomes a function of y. Such a function is called the inverse function of f and is denoted by f y1. We conclude that the inverse of f : D ™ R exists if and only if f is one-to-one. Definition 3.5.1. Let f : D ™ R. If there exists a function fy1 : f Ž D . ™ D such that fy1 w f Ž x .x s x and all xg D and f w fy1 Ž y .x s y for all y g f Ž D ., I then f y1 is called the inverse function of f. Definition 3.5.2. Let f : D ™ R. Then, f is said to be monotone increasing wdecreasing x on D if whenever x 1 , x 2 g D are such that x 1 - x 2 , then f Ž x 1 . F f Ž x 2 . w f Ž x 1 . G f Ž x 2 .x. The function f is strictly monotone increasing wdecreasing x on D if f Ž x 1 . - f Ž x 2 . w f Ž x 1 . ) f Ž x 2 .x whenever x 1 - x 2 . I If f is either monotone increasing or monotone decreasing on D, then it is called a monotone function on D. In particular, if it is either strictly monotone increasing or strictly monotone decreasing, then f Ž x . is strictly monotone on D. Strictly monotone functions have the property that their inverse functions exist. This will be shown in the next theorem. Theorem 3.5.1. Let f : D ™ R be strictly monotone increasing Žor decreasing. on D. Then, there exists a unique inverse function f y1 , which is strictly monotone increasing Žor decreasing . on f Ž D .. Proof. Let us suppose that f is strictly monotone increasing on D. To show that f y1 exists as a strictly monotone increasing function on f Ž D ..

77

INVERSE FUNCTIONS

Suppose that x 1 , x 2 g D are such that f Ž x 1 . s f Ž x 2 . s y. If x 1 / x 2 , then x 1 - x 2 or x 2 - x 1. Since f is strictly monotone increasing, then f Ž x 1 . - f Ž x 2 . or f Ž x 2 . - f Ž x 1 .. In either case, f Ž x 1 . / f Ž x 2 ., which contradicts the assumption that f Ž x 1 . s f Ž x 2 .. Hence, x 1 s x 2 , that is, f is one-to-one and has therefore a unique inverse f y1. The inverse f y1 is strictly monotone increasing on f Ž D .. To show this, suppose that f Ž x 1 . - f Ž x 2 .. Then, x 1 - x 2 . If not, we must have x 1 G x 2 . In this case, f Ž x 1 . s f Ž x 2 . when x 1 s x 2 , or f Ž x 1 . ) f Ž x 2 . when x 1 ) x 2 , since f is strictly monotone increasing. However, this is contrary to the assumption that f Ž x 1 . - f Ž x 2 .. Thus x 1 - x 2 and f y1 is strictly monotone increasing. The proof of Theorem 3.5.1 when ‘‘increasing’’ is replaced with ‘‘decreasing’’ is similar. I Theorem 3.5.2. Suppose that f : D ™ R is continuous and strictly monotone increasing Ždecreasing . on w a, b x ; D. Then, f y1 is continuous and strictly monotone increasing Ždecreasing . on f Žw a, b x.. Proof. By Theorem 3.5.1 we only need to show the continuity of f y1. Suppose that f is strictly monotone increasing. The proof when f is strictly monotone decreasing is similar. Since f is continuous on a closed and bounded interval, then by Corollary 3.4.1 it must achieve its infimum and supremum on w a, b x. Furthermore, because f is strictly monotone increasing, its infimum and supremum must be attained at only a and b, respectively. Thus f Ž w a, b x . s f Ž a . , f Ž b . . Let dg w f Ž a., f Ž b .x. There exists a unique value c, aF c F b, such that Ž f c . s d. For any ⑀ ) 0, let ␶ be defined as

␶ s min f Ž c . y f Ž c y ⑀ . , f Ž c q ⑀ . y f Ž c . . Then there exists a ␦ , 0 - ␦ - ␶ , such that all the x’s in w a, b x that satisfy the inequality f Ž x. yd -␦ must also satisfy the inequality < xy c < - ⑀ . This is true because f Ž c . y f Ž c . q f Ž c y ⑀ . - dy ␦ - f Ž x . - dq ␦ -f Ž c. qf Ž c q⑀ . yf Ž c. ,

78

LIMITS AND CONTINUITY OF FUNCTIONS

that is, f Ž cy⑀ . -f Ž x. -f Ž cq⑀ . . Using the fact that f we conclude that

y1

is strictly monotone increasing Žby Theorem 3.5.1.,

c y ⑀ - x- c q ⑀ , that is, < xy c < - ⑀ . It follows that xs fy1 Ž y . is continuous on w f Ž a., f Ž b .x. I Note that in general if y s f Ž x ., the equation y y f Ž x . s 0 may not produce a unique solution for x in terms of y. If, however, the domain of f can be partitioned into subdomains on each of which f is strictly monotone, then f can have an inverse on each of these subdomains. EXAMPLE 3.5.1. Consider the function f : R ™ R defined by y s f Ž x . s x 3. It is easy to see that f is strictly monotone increasing for all xg R. It therefore has a unique inverse given by fy1 Ž y . s y 1r3 . EXAMPLE 3.5.2. Let f : wy1, 1x ™ R be such that y s f Ž x . s x 5 y x. From Figure 3.2 it can be seen that f is strictly monotone increasing on D 1 s wy1,y 5y1r4 x and D 2 s w5y1r4 , 1x, but is strictly monotone decreasing on D 3 s wy5y1r4 , 5y1r4 x. This function has therefore three inverses, one on each of D 1 , D 2 , and D 3 . By Theorem 3.5.2, all three inverses are continuous. EXAMPLE 3.5.3. Let f : R ™ wy1, 1x be the function y s f Ž x . s sin x, where x is measured in radians. There is no unique inverse on R, since the sine function is not strictly monotone there. If, however, we restrict the domain of f to D s wy␲r2, ␲r2x, then f is strictly monotone increasing there and has the unique inverse f y1 Ž y . s Arcsin y Žsee Example 1.3.4.. The inverse of f on w␲r2, 3␲r2x is given by f y1 Ž y . s ␲ y Arcsin y. We can similarly find the inverse of f on w3␲r2, 5␲r2x, w5␲r2, 7␲r2x, etc.

Figure 3.2. The graph of the function f Ž x . s x 5 y x.

79

CONVEX FUNCTIONS

3.6. CONVEX FUNCTIONS Convex functions are frequently used in operations research. They also happen to be continuous, as will be shown in this section. The natural domains for such functions are convex sets. Definition 3.6.1. A set D ; R is convex if ␭ x 1 q Ž1 y ␭. x 2 g D whenever x 1 , x 2 belong to D and 0 F ␭ F 1. Geometrically, a convex set contains the line segment connecting any two of its points. The same definition actually applies to convex sets in R n, the n-dimensional Euclidean space Ž n G 2.. For example, each of the following sets is convex: 1. Any interval in R. 2. Any sphere in R 3, and in general, any hypersphere in R n, n G 4. 3. The set
Ž x, y . g R 2 < < x < q < y < F 14 . See Figure 3.3. I Definition 3.6.2.

A function f : D ™ R is convex if

f ␭ x1 q Ž 1 y ␭. x 2 F ␭ f Ž x1 . q Ž 1 y ␭. f Ž x 2 .

Ž 3.19 .

for all x 1 , x 2 g D and any ␭ such that 0 F ␭ F 1. The function f is strictly convex if inequality Ž3.19. is strict for x 1 / x 2 . Geometrically, inequality Ž3.19. means that if P and Q are any two points on the graph of y s f Ž x ., then the portion of the graph between P and Q lies below the chord PQ Žsee Figure 3.4.. Examples of convex functions include f Ž x . s x 2 on R, f Ž x . s sin x on w␲ , 2␲ x, f Ž x . s e x on R, f Ž x . s ylog x for x ) 0, to name just a few. I Definition 3.6.3.

A function f : D ™ R is concave if yf is convex.

I

We note that if f : w a, b x ™ R is convex and the values of f at a and b are finite, then f Ž x . is bounded from above on w a, b x by Ms max
f Ž a., f Ž b .4 . This is true because if xg w a, b x, then xs ␭ aq Ž1 y ␭. b for some ␭ g w0, 1x,

Figure 3.3. The set
Ž x, y . g R 2 < < x < q < y < F 14.

80

LIMITS AND CONTINUITY OF FUNCTIONS

Figure 3.4. The graph of a convex function.

since w a, b x is a convex set. Hence, f Ž x . F ␭ f Ž a. q Ž 1 y ␭. f Ž b . F ␭ Mq Ž 1 y ␭ . Ms M. The function f Ž x . is also bounded from below. To show this, we first note that any xg w a, b x can be written as xs

aq b 2

qt,

where aq b

ay

F t F by

2

aq b 2

.

Now, f

ž

aq b 2

/

F

1 2

ž

f

aq b 2

/

qt q

1 2

f

ž

aq b 2

/

yt ,

Ž 3.20 .

since if Ž aq b .r2 q t belongs to w a, b x, then so does Ž aq b .r2 y t. From Ž3.20. we then have f

ž

aq b 2

/ ž

qt G2 f

aq b 2

/ ž yf

aq b 2

Since f

ž

aq b 2

/

y t F M,

then f

ž

aq b 2

/ ž

qt G2 f

aq b 2

/

y M,

/

yt .

81

CONVEX FUNCTIONS

that is, f Ž x . G m for all xg w a, b x, where

ž

ms2 f

aq b 2

/

y M.

Another interesting property of convex functions is given in Theorem 3.6.1. Theorem 3.6.1. Let f : D ™ R be a convex function, where D is an open interval. Then f is LipŽ K, 1. on any closed interval w a, b x contained in D, that is, f Ž x1 . y f Ž x 2 . F K < x1 y x 2 <

Ž 3.21 .

for all x 1 , x 2 g w a, b x. Proof. Consider the closed interval w ay ⑀ , bq ⑀ x, where ⑀ ) 0 is chosen so that this interval is contained in D. Let m⬘ and M⬘ be, respectively, the lower and upper bounds of f Žas was seen earlier. on w ay ⑀ , bq ⑀ x. Let x 1 , x 2 be any two distinct points in w a, b x. Define z1 and ␭ as z1 s x 2 q

␭s

⑀ Ž x 2 y x1 . , < x1 y x 2 <

< x1 y x 2 <

⑀ q < x1 y x 2 <

.

Then z1 g w ay ⑀ , bq ⑀ x. This is true because Ž x 1 y x 1 .r < x 1 y x 2 < is either equal to 1 or to y1. Since x 2 g w a, b x, then ay ⑀ F x 2 y ⑀ F x 2 q

⑀ Ž x 2 y x1 . F x 2 q ⑀ F bq ⑀ . < x1 y x 2 <

Furthermore, it can be verified that x 2 s ␭ z1 q Ž 1 y ␭ . x 1 . We then have f Ž x 2 . F ␭ f Ž z1 . q Ž 1 y ␭ . f Ž x 1 . s ␭ f Ž z1 . y f Ž x 1 . q f Ž x 1 . . Thus, f Ž x 2 . y f Ž x 1 . F ␭ f Ž z1 . y f Ž x 1 . F ␭w M⬘ y m⬘ x F

< x1 y x 2 <

⑀

Ž M⬘ y m⬘ . s K < x 1 y x 2 < ,

Ž 3.22 .

82

LIMITS AND CONTINUITY OF FUNCTIONS

where K s Ž M⬘ y m⬘.r⑀ . Since inequality Ž3.22. is true for any x 1 , x 2 g w a, b x, we must also have f Ž x1 . y f Ž x 2 . F K < x1 y x 2 < .

Ž 3.23 .

From inequalities Ž3.22. and Ž3.23. we conclude that f Ž x1 . y f Ž x 2 . F K < x1 y x 2 < for any x 1 , x 2 g w a, b x, which shows that f Ž x . is LipŽ K, 1. on w a, b x.

I

Using Theorem 3.6.1 it is easy to prove the following corollary: Corollary 3.6.1. Let f : D ™ R be a convex function, where D is an open interval. If w a, b x is any closed interval contained in D, then f Ž x . is uniformly continuous on w a, b x and is therefore continuous on D. Note that if f Ž x . is convex on Ž a, b ., then it does not have to be continuous at the end points of the interval. It is easy to see, for example, that the function f : wy1, 1x ™ R defined as

½

2 f Ž x. s x , 2,

y1 - x- 1, xs 1,y 1

is convex on Žy1, 1., but is discontinuous at xs y1, 1.

3.7. CONTINUOUS AND CONVEX FUNCTIONS IN STATISTICS The most vivid examples of continuous functions in statistics are perhaps the cumulative distribution functions of continuous random variables. If X is a continuous random variable, then its cumulative distribution function FŽ x. sPŽ XFx. is continuous on R. In this case,

ž

P Ž X s a . s lim F aq n™⬁

1 n

/ ž

y F ay

1 n

/

s 0,

that is, the distribution of X assigns a zero probability to any single value. This is a basic characteristic of continuous random variables. Examples of continuous distributions include the beta, Cauchy, chisquared, exponential, gamma, Laplace, logistic, lognormal, normal, t, uniform, and the Weibull distributions. Most of these distributions are described

CONTINUOUS AND CONVEX FUNCTIONS IN STATISTICS

83

in introductory statistics books. A detailed account of their properties and uses is given in the two books by Johnson and Kotz Ž1970a, 1970b.. It is interesting to note that if X is any random variable Žnot necessarily continuous., then its cumulative distribution function, F Ž x ., is right-continuous on R Žsee, for example, Harris, 1966, page 55.. This function is also monotone increasing on R. If F Ž x . is strictly monotone increasing, then by Theorem 3.5.1 it has a unique inverse F y1 Ž y .. In this case, if Y has the uniform distribution over the open interval Ž0, 1., then the random variable Fy1 Ž Y . has the cumulative distribution function F Ž x .. To show this, consider X s Fy1 Ž Y .. Then, P Ž X F x . s P Fy1 Ž Y . F x sP YFFŽ x. sFŽ x. . This result has an interesting application in sampling. If Y1 , Y2 , . . . , Yn form an independent random sample from the uniform distribution U Ž0, 1., then Fy1 Ž Y1 ., Fy1 Ž Y2 ., . . . , Fy1 Ž Yn . will form an independent sample from a distribution with the cumulative distribution function F Ž x .. In other words, samples from any distribution can be generated through sampling from the uniform distribution U Ž0, 1.. This result forms the cornerstone of Monte Carlo simulation in statistics. Such a method provides an artificial way of collecting ‘‘data.’’ There are situations where the actual taking of a physical sample is either impossible or too expensive. In such situations, useful information can often be derived from simulated sampling. Monte Carlo simulation is also used in the study of the relative performance of test statistics and parameter estimators when the data come from certain specified parent distributions. Another example of the use of continuous functions in statistics is in limit theory. For example, it is known that if
X n4⬁ns1 is a sequence of random variables that converges in probability to c, and if g Ž x . is a continuous function at x s c, then the random variable g Ž X n . converges in probability to g Ž c . as n ™ ⬁. By definition, a sequence of random variables
X n4⬁ns1 converges in probability to a constant c if for a given ⑀ ) 0, lim P Ž < X n y c < G ⑀ . s 0.

n™⬁

In particular, if
X n4⬁ns1 is a sequence of estimators of a parameter c, then X n is said to be a consistent estimator of c if X n converges in probability to c. For example, the sample mean Xn s

1 n

n

Ý Xi

is1

84

LIMITS AND CONTINUITY OF FUNCTIONS

of a sample of size n from a population with a finite mean ␮ is a consistent estimator of ␮ according to the law of large numbers Žsee, for example, Lindgren, 1976, page 155.. Other types of convergence in statistics can be found in standard mathematical statistics books Žsee the annotated bibliography.. Convex functions also play an important role in statistics, as can be seen from the following examples. If f Ž x . is a convex function and X is a random variable with a finite mean ␮ s E Ž X ., then E f Ž X . Gf EŽ X . . Equality holds if and only if X is constant with probability 1. This inequality is known as Jensen’s inequality. If f is strictly convex, the inequality is strict unless X is constant with probability 1. A proof of Jensen’s inequality is given in Section 6.7.4. See also Lehmann Ž1983, page 50.. Jensen’s inequality has useful applications in statistics. For example, it can be used to show that if x 1 , x 2 , . . . , x n are n positive scalars, then their arithmetic mean is greater than or equal to their geometric mean, which is n equal to Ž ⌸ is1 x i .1r n. This can be shown as follows: Consider the convex function f Ž x . s ylog x. Let X be a discrete random variable that takes the values x 1 , x 2 , . . . , x n with probabilities equal to 1rn, that is,

°1 ¢0

~n,

PŽ Xsx. s

xs x 1 , x 2 , . . . , x n otherwise.

Then, by Jensen’s inequality, E Ž ylog X . G ylog E Ž X . .

Ž 3.24 .

However, E Ž ylog X . s y

1 n

n

Ý log x i ,

Ž 3.25 .

is1

and ylog E Ž X . s ylog

ž

1 n

n

Ý xi

is1

/

.

Ž 3.26 .

85

CONTINUOUS AND CONVEX FUNCTIONS IN STATISTICS

By using Ž3.25. and Ž3.26. in Ž3.24. we get 1 n

n

Ý log x i F log

is1

ž

n

1

Ý xi

n

is1

/

,

or

ž / n

log

Ł xi

1rn

F log

is1

ž

1 n

n

Ý xi

is1

/

.

Since the logarithmic function is monotone increasing, we conclude that

ž / n

Ł xi

1rn

F

is1

1 n

n

Ý xi .

is1

Jensen’s inequality can also be used to show that the arithmetic mean is greater than or equal to the harmonic mean. This assertion can be shown as follows: Consider the function f Ž x . s xy1 , which is convex for x ) 0. If X is a random variable with P Ž X ) 0. s 1, then by Jensen’s inequality,

ž / 1

E

X

1

G

EŽ X .

Ž 3.27 .

.

In particular, if X has the discrete distribution described earlier, then E

ž / 1

X

s

1 n

n

1

is1

xi

Ý

and EŽ X . s

n

1 n

Ý xi .

is1

By substitution in Ž3.27. we get 1 n

n

1

is1

xi

Ý

G

ž

n

1 n

Ý xi

is1

/

y1

,

or 1 n

n

Ý xi G

is1

ž

1 n

n

1

is1

xi

Ý

/

y1

.

Ž 3.28 .

The quantity on the right of inequality Ž3.28. is the harmonic mean of x1, x 2 , . . . , x n.

86

LIMITS AND CONTINUITY OF FUNCTIONS

Another example of the use of convex functions in statistics is in the general theory of estimation. Let X 1 , X 2 , . . . , X n be a random sample of size n from a population whose distribution depends on an unknown parameter ␪ . Let ␻ Ž X 1 , X 2 , . . . , X n . be an estimator of ␪ . By definition, the loss function Lw ␪ , ␻ Ž X 1 , X 2 , . . . , X n .x is a nonnegative function that measures the loss incurred when ␪ is estimated by ␻ Ž X 1 , X 2 , . . . , X n .. The expected value Žmean. of the loss function is called the risk function, denoted by RŽ ␪ , ␻ ., that is, R Ž ␪ , ␻ . s E
L ␪ , ␻ Ž X1 , X 2 , . . . , X n .

4.

The loss function is taken to be a convex function of ␪ . Examples of loss functions include the squared error loss, L ␪ , ␻ Ž X1 , X 2 , . . . , X n . s ␪ y ␻ Ž X1 , X 2 , . . . , X n . , 2

and the absolute error loss, L ␪ , ␻ Ž X1 , X 2 , . . . , X n . s ␪ y ␻ Ž X1 , X 2 , . . . , X n . . The first loss function is strictly convex, whereas the second is convex, but not strictly convex. The goodness of an estimator of ␪ is judged on the basis of its risk function, assuming that a certain loss function has been selected. The smaller the risk, the more desirable the estimator. An estimator ␻ *Ž X 1 , X 2 , . . . , X n . is said to be admissible if there is no other estimator ␻ Ž X 1 , X 2 , . . . , X n . of ␪ such that R Ž ␪ , ␻ . F R Ž ␪ , ␻ *. for all ␪ g ⍀ Ž ⍀ is the parameter space., with strict inequality for at least one ␪ . An estimator ␻ 0 Ž X 1 , X 2 , . . . , X n . is said to be a minimax estimator if it minimizes the supremum Žwith respect to ␪ . of the risk function, that is, sup R Ž ␪ , ␻ 0 . F sup R Ž ␪ , ␻ . ,

␪g⍀

␪g⍀

where ␻ Ž X 1 , X 2 , . . . , X n . is any other estimator of ␪ . It should be noted that a minimax estimator may not be admissible. EXAMPLE 3.7.1. Let X 1 , X 2 , . . . , X 20 be a random sample of size 20 from the normal distribution N Ž ␪ , 1.,y ⬁ - ␪ - ⬁. Let ␻ 1Ž X 1 , X 2 , . . . , X 20 . s X 20 be the sample mean, and let ␻ 2 Ž X 1 , X 2 , . . . , X 20 . s 0. Then, using a squared

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

87

error loss function, RŽ ␪ , ␻1 . s E

Ž X 20 y ␪ .

RŽ ␪ , ␻2 . s E Ž0 y ␪ .

2

2

s Var Ž X 20 . s 201 ,

s␪ 2.

In this case, sup R Ž ␪ , ␻ 1 . s 201 , ␪

whereas sup R Ž ␪ , ␻ 2 . s ⬁. ␪

Thus ␻ 1 s X 20 is a better estimator than ␻ 2 s 0. It can be shown that X 20 is the minimax estimator of ␪ with respect to a squared error loss function. Note, however, that X 20 is not an admissible estimator, since RŽ ␪ , ␻1 . F RŽ ␪ , ␻2 . for ␪ G 20y1r2 or ␪ F y20y1r2 . However, for y20y1r2 - ␪ - 20y1r2 , RŽ ␪ , ␻2 . - RŽ ␪ , ␻1 . . FURTHER READING AND ANNOTATED BIBLIOGRAPHY Corwin, L. J., and R. H. Szczarba Ž1982.. Multi®ariable Calculus. Dekker, New York. Fisz, M. Ž1963.. Probability Theory and Mathematical Statistics, 3rd ed. Wiley, New York. ŽSome continuous distributions are described in Chap. 5. Limit theorems concerning sequences of random variables are discussed in Chap. 6.. Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. Hardy, G. H. Ž1955.. A Course of Pure Mathematics, 10th ed. The University Press, Cambridge, England. ŽSection 89 of Chap. 4 gives definitions concerning the o and O symbols introduced in Section 3.3 of this chapter. . Harris, B. Ž1966.. Theory of Probability. Addison-Wesley, Reading, Massachusetts. ŽSome continuous distributions are given in Section 3.5.. Henle, J. M., and E. M. Kleinberg Ž1979.. Infinitesimal Calculus. The MIT Press, Cambridge, Massachusetts. Hillier, F. S., and G. J. Lieberman Ž1967.. Introduction to Operations Research. Holden-Day, San Francisco. ŽConvex sets and functions are described in Appendix 1.. Hogg, R. V., and A. T. Craig Ž1965.. Introduction to Mathematical Statistics, 2nd ed. Macmillan, New York. ŽLoss and risk functions are discussed in Section 9.3.. Hyslop, J. M. Ž1954.. Infinite Series, 5th ed. Oliver and Boyd, Edinburgh, England. ŽChap. 1 gives definitions and summaries of results concerning the o, O notation..

88

LIMITS AND CONTINUITY OF FUNCTIONS

Johnson, N. L., and S. Kotz Ž1970a.. Continuous Uni®ariate Distributionsᎏ1. Houghton Mifflin, Boston. Johnson, N. L., and S. Kotz Ž1970b.. Continuous Uni®ariate Distributionsᎏ2. Houghton Mifflin, Boston. Lehmann, E. L. Ž1983.. Theory of Point Estimation. Wiley, New York. ŽSection 1.6 discusses convex functions and their uses as loss functions. . Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽThe concepts of loss and utility, or negative loss, used in statistical decision theory are discussed in Chap. 8.. Randles, R. H., and D. A. Wolfe Ž1979.. Introduction to the Theory of Nonparametric Statistics. Wiley, New York. ŽSome mathematical statistics results, including Jensen’s inequality, are given in the Appendix.. Rao, C. R. Ž1973.. Linear Statistical Inference and Its Applications, 2nd ed. Wiley, New York. Roberts, A. W., and D. E. Varberg Ž1973.. Con®ex Functions. Academic Press, New York. ŽThis is a handy reference book that contains all the central facts about convex functions. . Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts. ŽChap. 12 defines and provides discussions concerning admissible and minimax estimators.. Rudin, W. Ž1964.. Principles of Mathematical Analysis, 2nd ed. McGraw-Hill, New York. ŽLimits of functions and some properties of continuous functions are given in Chap. 4.. Sagan, H. Ž1974.. Ad®anced Calculus. Houghton Mifflin, Boston. Smith, D. E. Ž1958.. History of Mathematics, Vol. 1. Dover, New York.

EXERCISES In Mathematics 3.1. Determine if the following limits exist:

Ž a.

lim

xy 1

x™1

Ž b. Ž c.

x5y1

lim x sin x™0

x

,

ž /ž /

lim sin x™0

1

,

1 x

sin

1 x

,

89

EXERCISES

°x y 1 , 2

~

Ž d.

lim f Ž x . , where f Ž x . s

x™0

x) 0,

xy 1 x3y1

¢2Ž xy 1. ,

x- 0.

3.2. Show that (a) tan x 3 s oŽ x 2 . as x™ 0. (b) xs oŽ'x . as x™ 0, (c) O Ž1. s oŽ x . as x™ ⬁, (d) f Ž x . g Ž x . s xy1 q O Ž1. as x™ 0, where f Ž x . s xq oŽ x 2 ., g Ž x . s xy2 q O Ž xy1 .. 3.3. Determine where the following functions are continuous, and indicate the points of discontinuity Žif any.:

½

x sin Ž 1rx . , 0,

Ž a.

f Ž x. s

Ž b.

f Ž x. s

Ž c.

ym r n , f Ž x. s x 0,

½

½

x/ 0, xs 0,

Ž xy 1 . r Ž 2 y x .

1r2

,

1,

x / 2, x s 2,

x/ 0, xs 0,

where m and n are positive integers,

Ž d.

f Ž x. s

x4y2 x2q3 x3 y1

,

x / 1.

3.4. Show that f Ž x . is continuous at xs a if and only if it is both leftcontinuous and right-continuous at xs a. 3.5. Use Definition 3.4.1 to show that the function f Ž x. sx2 y1 is continuous at any point ag R. 3.6. For what values of x is f Ž x . s lim

n™⬁

continuous?

3nx 1 y nx

90

LIMITS AND CONTINUITY OF FUNCTIONS

3.7. Consider the function f Ž x. s

xy < x < x

,

y1 - x- 1,

x / 0.

Can f Ž x . be defined at xs 0 so that it will be continuous there? 3.8. Let f Ž x . be defined for all xg R and continuous at x s 0. Furthermore, f Ž aq b . s f Ž a . q f Ž b . , for all a, b in R. Show that f Ž x . is uniformly continuous everywhere in R. 3.9. Let f Ž x . be defined as f Ž x. s

½

2 xy 1, x 3 y 5 x 2 q 5,

0 F x F 1, 1 F x F 2.

Determine if f Ž x . is uniformly continuous on w0, 2x. 3.10. Show that f Ž x . s cos x is uniformly continuous on R. 3.11. Prove Theorem 3.4.1. 3.12. A function f : D ™ R is called upper semicontinuous at ag D if for a given ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x . - f Ž a. q ⑀ for all xg N␦ Ž a. l D. If the above inequality is replaced with f Ž x . ) f Ž a. y ⑀ , then f Ž x . is said to be lower semicontinuous. Show that if D is closed and bounded, then (a) f Ž x . is bounded from above on D if f Ž x . is upper semicontinuous. (b) f Ž x . is bounded from below on D if f Ž x . is lower semicontinuous. 3.13. Let f : w a, b x ™ R be continuous such that f Ž x . s 0 for every rational number in w a, b x. Show that f Ž x . s 0 for every x in w a, b x. 3.14. For what values of x does the function f Ž x . s 3 q < x y 1 < q < xq 1 < have a unique inverse?

91

EXERCISES

3.15. Let f : R ™ R be defined as f Ž x. s

½

x, 2 xy 1,

x F 1, x ) 1.

Find the inverse function f y1 , 3.16. Let f Ž x . s 2 x 2 y 8 xq 8. Find the inverse of f Ž x . for (a) xF y2, (b) x) 2. 3.17. Suppose that f : w a, b x ™ R is a convex function. Show that for a given ⑀ ) 0 there exists a ␦ ) 0 such that n

Ý f Ž ai . y f Ž bi .

-⑀

is1

n for every finite, pairwise disjoint family of open subintervals
Ž a i , bi .4is1 n of w a, b x for which Ý is1 Ž bi y a i . - ␦ . Note: A function satisfying this property is said to be absolutely continuous on w a, b x.

3.18. Let f : w a, b x ™ R be a convex function. Show that if a1 , a2 , . . . , a n are positive numbers and x 1 , x 2 , . . . , x n are points in w a, b x, then f

ž

Ý nis1 a i x i A

/

F

Ý nis1 a i f Ž x i . A

,

where A s Ý nis1 a i . 3.19. Let f Ž x . be continuous on D ; R. Let S be the set of all xg D such that f Ž x . s 0. Show that S is a closed set. 3.20. Let f Ž x . be a convex function on D ; R. Show that expw f Ž x .x is also convex on D. In Statistics 3.21. Let X be a continuous random variable with the cumulative distribution function F Ž x . s 1 y eyx r ␪ ,

x) 0.

This is known as the exponential distribution. Its mean and variance are ␮ s ␪ , ␴ 2 s ␪ 2 , respectively. Generate a random sample of five observations from an exponential distribution with mean 2.

92

LIMITS AND CONTINUITY OF FUNCTIONS

w Hint: Select a ten-digit number from the table of random numbers, for example, 8389611097. Divide it by 10 10 to obtain the decimal number 0.8389611097. This number can be regarded as an observation from the uniform distribution U Ž0, 1.. Now, solve the equation F Ž x . s 0.8389611097. The resulting value of x is considered as an observation from the prescribed exponential distribution. Repeat this process four more times, each time selecting a new decimal number from the table of random numbers.x 3.22. Verify Jensen’s inequality in each of the following two cases: (a) X is normally distributed and f Ž x . s < x < . (b) X has the exponential distribution and f Ž x . s eyx . 3.23. Use the definition of convergence in probability to verify that if the sequence of random variables
X n4⬁ns1 converges in probability to zero, then so does the sequence
X n2 4⬁ns1 . 3.24. Show that EŽ X 2 . G EŽ < X< . . 2

w Hint: Let Y s < X < . Apply Jensen’s inequality to Y with f Ž x . s x 2 .x Deduce that if X has a mean ␮ and a variance ␴ 2 , then EŽ < Xy␮< . F␴ . 3.25. Consider the exponential distribution described in Exercise 3.21. Let X 1 , X 2 , . . . , X n be a sample of size n from this distribution. Consider the following estimators of ␪ : (a) ␻ 1Ž X 1 , X 2 , . . . , X n . s X n , the sample mean. (b) ␻ 2 Ž X 1 , X 2 , . . . , X n . s X n q 1, (c ) ␻ 3 Ž X 1 , X 2 , . . . , X n . s X n . Determine the risk function corresponding to a squared error loss function for each one of these estimators. Which estimator has the smallest risk for all values of ␪ ?

CHAPTER 4

Differentiation

Differentiation originated in connection with the problems of drawing tangents to curves and of finding maxima and minima of functions. Pierre de Fermat Ž1601᎐1665., the founder of the modern theory of numbers, is credited with having put forth the main ideas on which differential calculus is based. In this chapter, we shall introduce the notion of differentiation and study its applications in the determination of maxima and minima of functions. We shall restrict our attention to real-valued functions defined on R, the set of real numbers. The study of differentiation in connection with multivariable functions, that is, functions defined on R n Ž n G 1., will be considered in Chapter 7.

4.1. THE DERIVATIVE OF A FUNCTION The notion of differentiation was motivated by the need to find the tangent to a curve at a given point. Fermat’s approach to this problem was inspired by a geometric reasoning. His method uses the idea of a tangent as the limiting position of a secant when two of its points of intersection with the curve tend to coincide. This has lead to the modern notation associated with the derivative of a function, which we now introduce. Definition 4.1.1. Let f Ž x . be a function defined in a neighborhood Nr Ž x 0 . of a point x 0 . Consider the ratio

␾ Ž h. s

f Ž x0 q h. y f Ž x0 . h

,

Ž 4.1 .

where h is a nonzero increment of x 0 such that yr - h - r. If ␾ Ž h. has a limit as h ™ 0, then this limit is called the derivative of f Ž x . at x 0 and is 93

94

DIFFERENTIATION

denoted by f ⬘Ž x 0 .. It is also common to use the notation df Ž x . dx

xsx 0

s f ⬘Ž x0 . .

We thus have f ⬘ Ž x 0 . s lim

f Ž x0 q h. y f Ž x0 .

h™0

h

Ž 4.2 .

.

By putting xs x 0 q h, formula Ž4.2. can be written as f ⬘ Ž x 0 . s lim

x™x 0

f Ž x . y f Ž x0 . xy x 0

.

If f ⬘Ž x 0 . exists, then f Ž x . is said to be differentiable at x s x 0 . Geometrically, f ⬘Ž x 0 . is the slope of the tangent to the graph of the function y s f Ž x . at the point Ž x 0 , y 0 ., where y 0 s f Ž x 0 .. If f Ž x . has a derivative at every point of a set D, then f Ž x . is said to be differentiable on D. It is important to note that in order for f ⬘Ž x 0 . to exist, the left-sided and right-sided limits of ␾ Ž h. in formula Ž4.1. must exist and be equal as h ™ 0, or as x approaches x 0 from either side. It is possible to consider only one-sided derivatives at xs x 0 . These occur when ␾ Ž h. has just a one-sided limit as h ™ 0. We shall not, however, concern ourselves with such derivatives in this chapter. I Functions that are differentiable at a point must necessarily be continuous there. This will be shown in the next theorem. Theorem 4.1.1. Let f Ž x . be defined in a neighborhood of a point x 0 . If f Ž x . has a derivative at x 0 , then it must be continuous at x 0 . Proof. From Definition 4.1.1 we can write f Ž x 0 q h . y f Ž x 0 . s h␾ Ž h . . If the derivative of f Ž x . exists at x 0 , then ␾ Ž h. ™ f ⬘Ž x 0 . as h ™ 0. It follows from Theorem 3.2.1Ž2. that f Ž x0 q h. y f Ž x0 . ™ 0 as h ™ 0. Thus for a given ⑀ ) 0 there exists a ␦ ) 0 such that f Ž x0 q h. y f Ž x0 . - ⑀ if < h < - ␦ . This indicates that f Ž x . is continuous at x 0 .

I

95

THE DERIVATIVE OF A FUNCTION

It should be noted that even though continuity is a necessary condition for differentiability, it is not a sufficient condition, as can be seen from the following example: Let f Ž x . be defined as

° ¢0,

1

x sin , f Ž x . s~ x

x/ 0, xs 0.

This function is continuous at xs 0, since f Ž0. s lim x ™ 0 f Ž x . s 0 by the fact that x sin

1 x

F < x<

for all x. However, f Ž x . is not differentiable at xs 0. This is because when x s 0,

␾ Ž h. s

f Ž h . y f Ž 0. h h sin

s s sin

1 h

1 h h

y0 ,

since h / 0,

,

which does not have a limit as h ™ 0. Hence, f ⬘Ž0. does not exist. If f Ž x . is differentiable on a set D, then f ⬘Ž x . is a function defined on D. In the event f ⬘Ž x . itself is differentiable on D, then its derivative is called the second derivative of f Ž x . and is denoted by f ⬙ Ž x .. It is also common to use the notation d2 f Ž x . dx 2

sf⬙ Ž x. .

By the same token, we can define the nth Ž n G 2. derivative of f Ž x . as the derivative of the Ž n y 1.st derivative of f Ž x .. We denote this derivative by dnf Ž x . dx n

s f Ž n. Ž x . ,

n s 2, 3, . . . .

We shall now discuss some rules that pertain to differentiation. The reader is expected to know how to differentiate certain elementary functions such as polynomial, exponential, and trigonometric functions.

96

DIFFERENTIATION

Theorem 4.1.2. set D. Then

Let f Ž x . and g Ž x . be defined and differentiable on a

1. w ␣ f Ž x . q ␤ g Ž x .x⬘ s ␣ f ⬘Ž x . q ␤ g ⬘Ž x ., where ␣ and ␤ are constants. 2. w f Ž x . g Ž x .x⬘ s f ⬘Ž x . g Ž x . q f Ž x . g ⬘Ž x .. 3. w f Ž x .rg Ž x .x⬘ s w f ⬘Ž x . g Ž x . y f Ž x . g ⬘Ž x .xrg 2 Ž x . if g Ž x . / 0. Proof. The proof of Ž1. is straightforward. To prove Ž2. we write lim

f Ž xq h . g Ž xq h . y f Ž x . g Ž x . h

h™0

s lim

f Ž xq h . y f Ž x . g Ž xq h . q f Ž x . g Ž x q h . y g Ž x . h

h™0

s lim g Ž xq h . lim h™0

f Ž xq h . y f Ž x . h

h™0

q f Ž x . lim

g Ž xq h . y g Ž x . h

h™0

.

However, lim h ™ 0 g Ž xq h. s g Ž x ., since g Ž x . is continuous Žbecause it is differentiable .. Hence, lim

f Ž xq h . g Ž x q h . y f Ž x . g Ž x . h

h™0

s g Ž x . f ⬘Ž x . q f Ž x . g ⬘Ž x . .

Now, to prove Ž3. we write lim

f Ž xq h . rg Ž xq h . y f Ž x . rg Ž x . h

h™0

s lim h™0

s lim

g Ž x . f Ž xq h . y f Ž x . g Ž x q h . hg Ž x . g Ž x q h . g Ž x . f Ž xq h . y f Ž x . y f Ž x . g Ž x q h . y g Ž x . hg Ž x . g Ž x q h .

h™0

s

lim h ™ 0
g Ž x . f Ž xq h . y f Ž x . rh y f Ž x . g Ž xq h . y g Ž x . rh4

s

g Ž x . f ⬘Ž x . y f Ž x . g ⬘Ž x .

g Ž x . lim h ™ 0 g Ž xq h . g2Ž x.

.

I

97

THE DERIVATIVE OF A FUNCTION

Theorem 4.1.3 ŽThe Chain Rule.. Let f : D 1 ™ R and g: D 2 ™ R be two functions. Suppose that f Ž D 1 . ; D 2 . If f Ž x . is differentiable on D 1 and g Ž x . is differentiable on D 2 , then the composite function hŽ x . s g w f Ž x .x is differentiable on D 1 , and dg f Ž x .

s

dx

dg f Ž x .

df Ž x .

df Ž x .

dx

.

Proof. Let z s f Ž x . and t s f Ž xq h.. By the fact that g Ž z . is differentiable we can write g f Ž xq h . y g f Ž x . s g Ž t . y g Ž z . s Ž t y z . g ⬘Ž z . q o Ž t y z . ,

Ž 4.3 .

where, if we recall, the o-notation was introduced in Section 3.3. We then have g f Ž xq h . y g f Ž x .

s

h

tyz h

g ⬘Ž z . q

oŽ t y z . t y z ⭈ . tyz h

Ž 4.4 .

As h ™ 0, t ™ z, and hence lim h™0

tyz h

s lim

f Ž xq h . y f Ž x . h

h™0

s

df Ž x . dx

.

Now, by taking the limits of both sides of Ž4.4. as h ™ 0 and noting that lim h™0

oŽ t y z . tyz

s lim

oŽ t y z .

t™z

tyz

s 0,

we conclude that dg f Ž x . dx

s

df Ž x . dg f Ž x . df Ž x .

dx

.

I

NOTE 4.1.1. We recall that f Ž x . must be continuous in order for f ⬘Ž x . to exist. However, if f ⬘Ž x . exists, it does not have to be continuous. Care should be exercised when showing that f ⬘Ž x . is continuous. For example, let us consider the function

° sin 1 , ¢0, x

~x

f Ž x. s

2

x/ 0, xs 0.

98

DIFFERENTIATION

Suppose that it is desired to show that f ⬘Ž x . exists, and if so, to determine if it is continuous. To do so, let us first find out if f ⬘Ž x . exists at xs 0: f Ž h . y f Ž 0.

f ⬘ Ž 0 . s lim

h

h™0

h2 sin s lim

1 h

h

h™0

s lim h sin h™0

1 h

s 0.

Thus the derivative of f Ž x . exists at xs 0 and is equal to zero. For x / 0, it is clear that the derivative of f Ž x . exists. By applying Theorem 4.1.2 and using our knowledge of the derivatives of elementary functions, f ⬘Ž x . can be written as

° ¢0,

1

1

~2 x sin x y cos x ,

f ⬘Ž x . s

x / 0, xs 0.

We note that f ⬘Ž x . exists for all x, but is not continuous at xs 0, since

ž

lim f ⬘ Ž x . s lim 2 x sin

x™0

x™0

1 x

y cos

1 x

/

does not exist, because cosŽ1rx . has no limit as x ™ 0. However, for any nonzero value of x, f ⬘Ž x . is continuous. If f Ž x . is a convex function, then we have the following interesting result, whose proof can be found in Roberts and Varberg Ž1973, Theorem C, page 7.: Theorem 4.1.4. If f : Ž a, b . ™ R is convex on the open interval Ž a, b ., then the set S where f ⬘Ž x . fails to exist is either finite or countable. Moreover, f ⬘Ž x . is continuous on Ž a, b . y S, the complement of S with respect to Ž a, b .. For example, the function f Ž x . s < x < is convex on R. Its derivatives does not exist at xs 0 Žwhy?., but is continuous everywhere else. The sign of f ⬘Ž x . provides information about the behavior of f Ž x . in a neighborhood of x. More specifically, we have the following theorem: Theorem 4.1.5. Let f : D ™ R, where D is an open set. Suppose that f ⬘Ž x . is positive at a point x 0 g D. Then there is a neighborhood N␦ Ž x 0 . ; D such that for each x in this neighborhood, f Ž x . ) f Ž x 0 . if x ) x 0 , and f Ž x . - f Ž x 0 . if x- x 0 .

99

THE MEAN VALUE THEOREM

Proof. Let ⑀ s f ⬘Ž x 0 .r2. Then, there exists a ␦ ) 0 such that f ⬘Ž x0 . y ⑀ -

f Ž x . y f Ž x0 . xy x 0

- f ⬘Ž x0 . q ⑀

if < xy x 0 < - ␦ . Hence, if x) x 0 , f Ž x . y f Ž x0 . )

Ž xy x 0 . f ⬘ Ž x 0 . 2

,

which shows that f Ž x . ) f Ž x 0 . since f ⬘Ž x 0 . ) 0. Furthermore, since f Ž x . y f Ž x0 . xy x 0 then f Ž x . - f Ž x 0 . if x- x 0 .

) 0,

I

If f ⬘Ž x 0 . - 0, it can be similarly shown that f Ž x . - f Ž x 0 . if x) x 0 , and f Ž x . ) f Ž x 0 . if x- x 0 . 4.2. THE MEAN VALUE THEOREM This is one of the most important theorems in differential calculus. It is also known as the theorem of the mean. Before proving the mean value theorem, let us prove a special case of it known as Rolle’s theorem. Theorem 4.2.1 ŽRolle’s Theorem.. Let f Ž x . be continuous on the closed interval w a, b x and differentiable on the open interval Ž a, b .. If f Ž a. s f Ž b ., then there exists a point c, a- c - b, such that f ⬘Ž c . s 0. Proof. Let d denote the common value of f Ž a. and f Ž b .. Define hŽ x . s Ž f x . y d. Then hŽ a. s hŽ b . s 0. If hŽ x . is also zero on Ž a, b ., then h⬘Ž x . s 0 for a- x- b and the theorem is proved. Let us therefore assume that hŽ x . / 0 for some xg Ž a, b .. Since hŽ x . is continuous on w a, b x wbecause f Ž x . isx, then by Corollary 3.4.1 it must achieve its supremum M at a point ␰ in w a, b x, and its infimum m at a point ␩ in w a, b x. If hŽ x . ) 0 for some x g Ž a, b ., then we must obviously have a- ␰ - b, because hŽ x . vanishes at both end points. We now claim that h⬘Ž ␰ . s 0. If h⬘Ž ␰ . ) 0 or - 0, then by Theorem 4.1.5, there exists a point x 1 in a neighborhood N␦ 1Ž ␰ . ; Ž a, b . at which hŽ x 1 . ) hŽ ␰ ., a contradiction, since hŽ ␰ . s M. Thus h⬘Ž ␰ . s 0, which implies that f ⬘Ž ␰ . s 0, since h⬘Ž x . s f ⬘Ž x . for all x g Ž a, b .. We can similarly arrive at the conclusion that f ⬘Ž␩ . s 0 if hŽ x . - 0 for some xg Ž a, b .. In this case, if h⬘Ž␩ . / 0, then by Theorem 4.1.5 there exists a point x 2 in a neigh-

100

DIFFERENTIATION

borhood N␦ 2Ž␩ . ; Ž a, b . at which hŽ x 2 . - hŽ␩ . s m, a contradiction, since m is the infimum of hŽ x . over w a, b x. Thus in both cases, whether hŽ x . ) 0 or - 0 for some x g Ž a, b ., we must have a point c, a- c - b, such that f ⬘Ž c . s 0. I Rolle’s theorem has the following geometric interpretation: If f Ž x . satisfies the conditions of Theorem 4.2.1, then the graph of y s f Ž x . must have a tangent line that is parallel to the x-axis at some point c between a and b. Note that there can be several points like c inside Ž a, b .. For example, the function y s x 3 y 5 x 2 q 3 xy 1 satisfies the conditions of Rolle’s theorem on the interval w a, b x, where as 0 and bs Ž5 q '13 .r2. In this case, f Ž a. s f Ž b . s y1, and f ⬘Ž x . s 3 x 2 y 10 xq 3 vanishes at x s 13 and x s 3. Theorem 4.2.2 ŽThe Mean Value Theorem.. If f Ž x . is continuous on the closed interval w a, b x and differentiable on the open interval Ž a, b ., then there exists a point c, a- c - b, such that f Ž b . s f Ž a . q Ž by a . f ⬘ Ž c . . Proof. Consider the function ⌽ Ž x . s f Ž x . y f Ž a . y A Ž xy a. , where As

f Ž b . y f Ž a. by a

.

The function ⌽ Ž x . is continuous on w a, b x and is differentiable on Ž a, b ., since ⌽⬘Ž x . s f ⬘Ž x . y A. Furthermore, ⌽ Ž a. s ⌽ Ž b . s 0. It follows from Rolle’s theorem that there exists a point c, a- c - b, such that ⌽⬘Ž c . s 0. Thus, f ⬘Ž c . s which proves the theorem.

f Ž b . y f Ž a. by a

,

I

The mean value theorem has also a nice geometric interpretation. If the graph of the function y s f Ž x . has a tangent line at each point of its length between two points P1 and P2 Žsee Figure 4.1., then there must be a point Q between P1 and P2 at which the tangent line is parallel to the secant line through P1 and P2 . Note that there can be several points on the curve between P1 and P2 that have the same property as Q, as can be seen from Figure 4.1. The mean value theorem is useful in the derivation of several interesting results, as will be seen in the remainder of this chapter.

101

THE MEAN VALUE THEOREM

Figure 4.1. Tangent lines parallel to the secant line.

Corollary 4.2.1. If f Ž x . has a derivative f ⬘Ž x . that is nonnegative Žnonpositive. on an interval Ž a, b ., then f Ž x . is monotone increasing Ždecreasing . on Ž a, b .. If f ⬘Ž x . is positive Žnegative. on Ž a, b ., then f Ž x . is strictly monotone increasing Ždecreasing . there. Proof. Let x 1 and x 2 be two points in Ž a, b . such that x 1 - x 2 . By the mean value theorem, there exists a point x 0 , x 1 - x 0 - x 2 , such that f Ž x 2 . s f Ž x1 . q Ž x 2 y x1 . f ⬘ Ž x 0 . . If f ⬘Ž x 0 . G 0, then f Ž x 2 . G f Ž x 1 . and f Ž x . is monotone increasing. Similarly, if f ⬘Ž x 0 . F 0, then f Ž x 2 . F f Ž x 1 . and f Ž x . is monotone decreasing. If, however, f ⬘Ž x . ) 0, or f ⬘Ž x . - 0 on Ž a, b ., then strict monotonicity follows over Ž a, b .. I Theorem 4.2.3. If f Ž x . is monotone increasing wdecreasing x on an interval Ž a, b ., and if f Ž x . is differentiable there, then f ⬘Ž x . G 0 w f ⬘Ž x . F 0x on Ž a, b .. Proof. Let x 0 g Ž a, b .. There exists a neighborhood Nr Ž x 0 . ; Ž a, b .. Then, for any xg Nr Ž x 0 . such that x/ x 0 , the ratio f Ž x . y f Ž x0 . xy x 0 is nonnegative. This is true because f Ž x . G f Ž x 0 . if x) x 0 and f Ž x . F f Ž x 0 . if x- x 0 . By taking the limit of this ratio as x™ x 0 , we claim that f ⬘Ž x 0 . G 0. To prove this claim, suppose that f ⬘Ž x 0 . - 0. Then there exists a ␦ ) 0 such that f Ž x . y f Ž x0 . xy x 0

y f ⬘Ž x0 . - y

1 2

f ⬘Ž x0 .

102

DIFFERENTIATION

if < xy x 0 < - ␦ . It follows that f Ž x . y f Ž x0 . xy x 0

-

1 2

f ⬘ Ž x 0 . - 0.

Thus f Ž x . - f Ž x 0 . if x) x 0 , which is a contradiction. Hence, f ⬘Ž x 0 . G 0. A similar argument can be used to show that f ⬘Ž x 0 . F 0 when f Ž x . is monotone decreasing. I Note that strict monotonicity on a set D does not necessarily imply that f ⬘Ž x . ) 0, or f ⬘Ž x . - 0, for all x in D. For example, the function f Ž x . s x 3 is strictly monotone increasing for all x, but f ⬘Ž0. s 0. We recall from Theorem 3.5.1 that strict monotonicity of f Ž x . is a sufficient condition for the existence of the inverse function fy1. The next theorem shows that under certain conditions, fy1 is a differentiable function. Theorem 4.2.4. Suppose that f Ž x . is strictly monotone increasing Žor decreasing . and continuous on an interval w a, b x. If f ⬘Ž x . exists and is different from zero at x 0 g Ž a, b ., then the inverse function fy1 is differentiable at y 0 s f Ž x 0 . and its derivative is given by dfy1 Ž y .

s

dy

ysy 0

1 f ⬘Ž x0 .

.

Proof. By Theorem 3.5.2, fy1 Ž y . exists and is continuous. Let x 0 g Ž a, b ., and let Nr Ž x 0 . ; Ž a, b . for some r ) 0. Then, for any x g Nr Ž x 0 ., fy1 Ž y . y fy1 Ž y 0 . y y y0

s

x y x0 f Ž x . y f Ž x0 .

s

1 f Ž x . y f Ž x 0 . r Ž xy x 0 .

,

Ž 4.5 .

where y s f Ž x .. Now, since both f and fy1 are continuous, then x ™ x 0 if and only if y ™ y 0 . By taking the limits of all sides in formula Ž4.5., we conclude that the derivative of fy1 at y 0 exists and is equal to dfy1 Ž y . dy

s ysy 0

1 f ⬘Ž x0 .

.

I

The following theorem gives a more general version of the mean value theorem. It is due to Augustin-Louis Cauchy and has an important application in calculating certain limits, as will be seen later in this chapter.

103

THE MEAN VALUE THEOREM

Theorem 4.2.5 ŽCauchy’s Mean Value Theorem.. If f Ž x . and g Ž x . are continuous on the closed interval w a, b x and differentiable on the open interval Ž a, b ., then there exists a point c, a- c - b, such that f Ž b . y f Ž a. g ⬘ Ž c . s g Ž b . y g Ž a. f ⬘ Ž c . , Proof. The proof is based on using Rolle’s theorem in a manner similar to that of Theorem 4.2.2. Define the function ␺ Ž x . as

␺ Ž x . s f Ž b. y f Ž x .

g Ž b . y g Ž a. y f Ž b . y f Ž a.

g Ž b. y g Ž x . .

This function is continuous on w a, b x and is differentiable on Ž a, b ., since

␺ ⬘ Ž x . s yf ⬘ Ž x . g Ž b . y g Ž a . q g ⬘ Ž x . f Ž b . y f Ž a . . Furthermore, ␺ Ž a. s ␺ Ž b . s 0. Thus by Rolle’s theorem, there exists a point c, a- c - b, such that ␺ ⬘Ž c . s 0, that is, yf ⬘ Ž c . g Ž b . y g Ž a . q g ⬘ Ž c . f Ž b . y f Ž a . s 0.

Ž 4.6 .

In particular, if g Ž b . y g Ž a. / 0 and f ⬘Ž x . and g ⬘Ž x . do not vanish at the same point in Ž a, b ., then formula Ž4.6. an be written as f ⬘Ž c . g ⬘Ž c .

s

f Ž b . y f Ž a. g Ž b . y g Ž a.

I

.

An immediate application of this theorem is a very popular method in calculating the limits of certain ratios of functions. This method is attributed to Guillaume Francois Marquis de l’Hospital Ž1661᎐1704. and is known as l’Hospital’s rule. It deals with the limit of the ratio f Ž x .rg Ž x . as x™ a when both the numerator and the denominator tend simultaneously to zero or to infinity as x™ a. In either case, we have what is called an indeterminate ratio caused by having 0r0 or ⬁r⬁ as x™ a. Theorem 4.2.6 Žl’Hospital’s Rule.. Let f Ž x . and g Ž x . be continuous on the closed interval w a, b x and differentiable on the open interval Ž a, b .. Suppose that we have the following: 1. g Ž x . and g ⬘Ž x . are not zero at any point inside Ž a, b .. 2. lim x ™ aq f ⬘Ž x .rg ⬘Ž x . exists. 3. f Ž x . ™ 0 and g Ž x . ™ 0 as x™ aq, or f Ž x . ™ ⬁ and g Ž x . ™ ⬁ as x™ aq. Then, limq

x™a

f Ž x. gŽ x.

s limq x™a

f ⬘Ž x . g ⬘Ž x .

.

104

DIFFERENTIATION

Proof. For the sake of simplicity, we shall drop the q sign from aq and simply write x™ a when x approaches a from the right. Let us consider the following cases: CASE 1. f Ž x . ™ 0 and g Ž x . ™ 0 as x ™ a, where a is finite. Let x g Ž a, b .. By applying Cauchy’s mean value theorem on the interval w a, x x we get f Ž x. gŽ x.

f Ž x . y f Ž a.

s

s

g Ž x . y g Ž a.

f ⬘Ž c . g ⬘Ž c .

.

where a- c - x. Note that f Ž a. s g Ž a. s 0, since f Ž x . and g Ž x . are continuous and their limits are equal to zero when x™ a. Now, as x™ a, c ™ a; hence lim

x™a

CASE 2. Then

f Ž x. gŽ x.

s lim

c™a

f ⬘Ž c .

s lim

g ⬘Ž c .

x™a

f ⬘Ž x . g ⬘Ž x .

.

f Ž x . ™ 0 and g Ž x . ™ 0 as x ™ ⬁. Let zs 1rx. As x™ ⬁, z™ 0.

lim

x™⬁

f Ž x. gŽ x.

s lim z™0

f 1Ž z . g 1Ž z .

,

Ž 4.7 .

where f 1Ž z . s f Ž1rz . and g 1Ž z . s g Ž1rz .. These functions are continuous since f Ž x . and g Ž x . are, and z/ 0 as z™ 0 Žsee Theorem 3.4.2.. Here, we find it necessary to set f 1Ž0. s g 1Ž0. s 0 so that f 1Ž z . and g 1Ž z . will be continuous at z s 0, since their limits are equal to zero. This is equivalent to defining f Ž x . and g Ž x . to be zero at infinity in the extended real number system. Furthermore, by the chain rule of Theorem 4.1.3 we have

ž / Ž .ž / 1

f 1X Ž z . s f ⬘ Ž x . y g X1 Ž z . s g ⬘ x

z2

y

1

z2

, .

If we now apply Case 1, we get lim z™0

f 1Ž z . g 1Ž z .

s lim z™0

s lim

x™⬁

f 1X Ž z . g X1 Ž z . f ⬘Ž x . g ⬘Ž x .

.

Ž 4.8 .

105

THE MEAN VALUE THEOREM

From Ž4.7. and Ž4.8. we then conclude that lim

x™⬁

f Ž x. gŽ x.

f ⬘Ž x .

s lim

g ⬘Ž x .

x™⬁

.

CASE 3. f Ž x . ™ ⬁ and g Ž x . ™ ⬁ as x™ a, where a is finite. Let lim x ™ aw f ⬘Ž x .rg ⬘Ž x .x s L. Then for a given ⑀ ) 0 there exists a ␦ ) 0 such that aq ␦ - b and f ⬘Ž x . g ⬘Ž x .

Ž 4.9 .

yL -⑀ ,

if a- x- aq ␦ . By applying Cauchy’s mean value theorem on the interval w x,aq ␦ x we get f Ž x . y f Ž aq ␦ .

s

g Ž x . y g Ž aq ␦ .

f ⬘Ž d . g ⬘Ž d .

,

where x- d- aq ␦ . From inequality Ž4.9. we then have f Ž x . y f Ž aq ␦ . g Ž x . y g Ž aq ␦ .

yL -⑀

for all x such that a- x- aq ␦ . It follows that L s lim

x™a

s lim

x™a

s lim

x™a

f Ž x . y f Ž aq ␦ . g Ž x . y g Ž aq ␦ . f Ž x. gŽ x.

lim

x™a

1 y f Ž aq ␦ . rf Ž x . 1 y g Ž aq ␦ . rg Ž x .

f Ž x. gŽ x.

since both f Ž x . and g Ž x . tend to ⬁ as x™ a. CASE 4. f Ž x . ™ ⬁ and g Ž x . ™ ⬁ as x™ ⬁. This can be easily shown by using the techniques applied in Cases 2 and 3. CASE 5. lim x ™ a f ⬘Ž x .rg⬘Ž x . s ⬁, where a is finite or infinite. Let us consider the ratio g Ž x .rf Ž x .. We have that lim

x™a

g ⬘Ž x . f ⬘Ž x .

s 0.

106

DIFFERENTIATION

Hence, lim

x™a

gŽ x. f Ž x.

s lim

x™a

g ⬘Ž x . f ⬘Ž x .

s 0.

If A is any positive number, then there exists a ␦ ) 0 such that gŽ x.

-

f Ž x.

1 A

,

if a- x- aq ␦ . Thus for such values of x, f Ž x. gŽ x.

) A,

which implies that lim

x™a

f Ž x. gŽ x.

s ⬁.

When applying l’Hospital’s rule, the ratio f ⬘Ž x .rg ⬘Ž x . may assume the indeterminate from 0r0 or ⬁r⬁ as x™ a. In this case, higher-order derivatives of f Ž x . and g Ž x ., assuming such derivatives exist, will be needed. In general, if the first n y 1 Ž n G 1. derivatives of f Ž x . and g Ž x . tend simultaneously to zero or to ⬁ as x™ a, and if the nth derivatives, f Ž n. Ž x . and g Ž n. Ž x ., exist and satisfy the same conditions as those imposed on f ⬘Ž x . and g ⬘Ž x . in Theorem 4.2.6, then lim

x™a

f Ž x. gŽ x.

s lim

x™a

f Ž n. Ž x . g Ž n. Ž x .

I

.

A Historical Note According to Eves Ž1976, page 342., in 1696 the Marquis de l’Hospital assembled the lecture notes of his teacher Johann Bernoulli Ž1667᎐1748. into the world’s first textbook on calculus. In this book, the so-called l’Hospital’s rule is found. It is perhaps more accurate to refer to this rule as the Bernoulli-l’Hospital rule. Note that the name l’Hospital follows the old French spelling and the letter s is not to be pronounced. In modern French this name is spelled as l’Hopital. ˆ sin x

lim

s lim

cos x

s 1. x 1 x™0 This is a well-known limit. It implies that sin x and x are asymptotically equal, that is, sin x ; x as x ™ 0 Žsee Section 3.3.. EXAMPLE 4.2.1.

x™0

EXAMPLE 4.2.2.

lim

1 y cos x

s lim

sin x

s lim

cos x

s

1

. 2 2 x x™0 2 x x™0 We note here that l’Hospital’s rule was applied twice before reaching the limit 12 . x™0

2

107

THE MEAN VALUE THEOREM

a

x

, where a) 1. x This is of the form ⬁r⬁ as x™ ⬁. Since a x s e x log a, then EXAMPLE 4.2.3.

lim

x™⬁

lim

ax

x™⬁

x

e x log a Ž log a .

s lim

1

x™⬁

s ⬁.

This is also a well-known limit. On the basis of this result it can be shown that Žsee Exercise 4.12. the following hold: ax

s ⬁, where a) 1, m ) 0. xm log x 2. lim s 0, where m ) 0. m x™⬁ x 1. lim

x™⬁

EXAMPLE 4.2.4. lim x ™ 0q x x. This is of the form 0 0 as x™ 0q, which is indeterminate. It can be reduced to the form 0r0 or ⬁r⬁ so that l’Hospital’s rule can apply. To do so we write x x as x x s e x log x . However, x log xs

log x

,

1rx

which is of the form y⬁r⬁ as x™ 0q. By l’Hospital’s rule we then have limq Ž x log x . s limq

x™0

x™0

1rx y1rx 2

s limq Ž yx . x™0

s 0. It follows that lim x x s limq e x log

x™0 q

x

x™0

lim Ž x log x .

s exp

x™0 q

s 1. EXAMPLE 4.2.5.

lim x log

ž

xq 1

/

. xy 1 This is of the form ⬁ = 0 as x™ ⬁, which is indeterminate. But x™⬁

x log

ž

xq 1 xy 1

/

log s

ž

xq 1

xy 1 1rx

/

108

DIFFERENTIATION

is of the form 0r0 as x™ ⬁. Hence, y2 lim x log

x™⬁

ž

xq 1 xy 1

/

s lim

Ž xy 1 . Ž xq 1 . y1rx 2

x™⬁

s lim

x™⬁

2

Ž 1 y 1rx . Ž 1 q 1rx .

s 2. We can see from the foregoing examples that the use of l’Hospital’s rule can facilitate the process of finding the limit of the ratio f Ž x .rg Ž x . as x ™ a. In many cases, it is easier to work with f ⬘Ž x .rg ⬘Ž x . than with the original ratio. Many other indeterminate forms can also be resolved by l’Hospital’s rule by first reducing them to the form 0r0 or ⬁r⬁ as was shown in Examples 4.2.4 and 4.2.5. It is important here to remember that the application of l’Hospital’s rule requires that the limit of f ⬘Ž x .rg⬘Ž x . exist as a finite number or be equal to infinity in the extended real number system as x™ a. If this is not the case, then it does not necessarily follow that the limit of f Ž x .rg Ž x . does not exist. For example, consider f Ž x . s x 2 sinŽ1rx . and g Ž x . s x. Here, f Ž x .rg Ž x . tends to zero as x™ 0, as was seen earlier in this chapter. However, the ratio f ⬘Ž x . g ⬘Ž x .

s 2 x sin

1 x

y cos

1 x

has no limit as x™ 0, since it oscillates inside a small neighborhood of the origin.

4.3. TAYLOR’S THEOREM This theorem is also known as the general mean value theorem, since it is considered as an extension of the mean value theorem. It was formulated by the English mathematician Brook Taylor Ž1685᎐1731. in 1712 and has since become a very important theorem in calculus. Taylor used his theorem to expand functions into infinite series. However, full recognition of the importance of Taylor’s expansion was not realized until 1755, when Leonhard Euler Ž1707᎐1783. applied it in his differential calculus, and still later, when Joseph Louis Lagrange Ž1736᎐1813. used it as the foundation of his theory of functions. Theorem 4.3.1 ŽTaylor’s Theorem.. If the Ž n y 1.st Ž n G 1. derivative of Ž f x ., namely f Ž ny1. Ž x ., is continuous on the closed interval w a, b x and the nth derivative f Ž n. Ž x . exists on the open interval Ž a, b ., then for each xg w a, b x

TAYLOR’S THEOREM

109

we have

Ž xy a.

f Ž x . s f Ž a . q Ž xy a . f ⬘ Ž a . q

2

f ⬙ Ž a.

2!

n

Ž xy a. Ž xy a. Ž n. q ⭈⭈⭈ q f Ž ny1. Ž a . q f Ž ␰ ., n! Ž n y 1. ! ny1

where a- ␰ - x. Proof. The method to prove this theorem is very similar to the one used for Theorem 4.2.2. For a fixed x in w a, b x let the function ␺nŽ t . be defined as

␺n Ž t . s g n Ž t . y

ž

xy t xy a

/

n

g n Ž a. ,

where aF t F b and g n Ž t . s f Ž x . y f Ž t . y Ž xy t . f ⬘ Ž t . y

Ž xy t .

2

2!

Ž xy t . f ⬙ Ž t . y ⭈⭈⭈ y f Ž ny1. Ž t . . Ž n y 1. ! ny1

Ž 4.10 .

The function ␺nŽ t . has the following properties: 1. ␺nŽ a. s 0 and ␺nŽ x . s 0. 2. ␺nŽ t . is a continuous function of t on w a, b x. 3. The derivative of ␺nŽ t . with respect to t exists on Ž a, b .. This derivative is equal to

␺nX

Ž t.

s g Xn

Ž t. q

nŽ x y t .

ny1

Ž xy a.

n

g n Ž a.

s yf ⬘ Ž t . q f ⬘ Ž t . y Ž xy t . f ⬙ Ž t . q Ž xy t . f ⬙ Ž t .

Ž xy t . f Ž ny1. Ž t . Ž n y 2. ! ny2

y ⭈⭈⭈ q

nŽ x y t . Ž xy t . y f Ž n. Ž t . q g n Ž a. n Ž n y 1. ! Ž xy a. ny1

ny1

nŽ x y t . Ž xy t . sy f Ž n. Ž t . q g n Ž a. . n Ž n y 1. ! Ž xy a. ny1

ny1

110

DIFFERENTIATION

By applying Rolle’s theorem to ␺nŽ t . on the interval w a, x x we can assert that there exists a value ␰ , a- ␰ - x, such that ␺nX Ž ␰ . s 0, that is, nŽ x y ␰ . Ž xy ␰ . y f Ž n. Ž ␰ . q n Ž n y 1. ! Ž xy a. ny1

ny1

g n Ž a . s 0,

or g n Ž a. s

Ž xy a. n!

n

f Ž n. Ž ␰ . .

Ž 4.11 .

Using formula Ž4.10. in Ž4.11., we finally get

f Ž x . s f Ž a . q Ž xy a . f ⬘ Ž a . q

Ž xy a.

2

2!

f ⬙ Ž a. n

Ž xy a. Ž xy a. Ž n. q ⭈⭈⭈ q f Ž ny1. Ž a . q f Ž ␰ . . Ž 4.12 . n! Ž n y 1. ! ny1

This is known as Taylor’s formula. It can also be expressed as

f Ž aq h . s f Ž a . q hf ⬘ Ž a . q q ⭈⭈⭈ q

h ny1

Ž n y 1. !

h2 2!

f ⬙ Ž a.

f Ž ny1. Ž a . q

where h s xy a and 0 - ␪n - 1.

hn n!

f Ž n. Ž aq ␪n h . , Ž 4.13 .

I

In particular, if f Ž x . has derivatives of all orders in some neighborhood Nr Ž a. of the point a, formula Ž4.12. can provide a series expansion of f Ž x . for xg Nr Ž a. as n ™ ⬁. The last term in formula Ž4.12., or formula Ž4.13., is called the remainder of Taylor’s series and is denoted by R n . Thus,

Rn s s

Ž xy a. n! hn n!

n

f Ž n. Ž ␰ .

f Ž n. Ž aq ␪n h . .

TAYLOR’S THEOREM

111

If R n ™ 0 as n ™ ⬁, then

Ž xy a.

⬁

Ý

f Ž x . s f Ž a. q

n

n!

ns1

f Ž n. Ž a . ,

Ž 4.14 .

or ⬁

hn

ns1

n!

Ý

f Ž aq h . s f Ž a . q

f Ž n. Ž a . .

Ž 4.15 .

This results in what is known as Taylor’s series. Thus the validity of Taylor’s series is contingent on having R n ™ 0 as n ™ ⬁, and on having derivatives of all orders for f Ž x . in Nr Ž a.. The existence of these derivatives alone is not sufficient to guarantee a valid expansion. A special form of Taylor’s series is Maclaurin’s series, which results when as 0. Formula Ž4.14. then reduces to ⬁

xn

ns1

n!

Ý

f Ž x . s f Ž 0. q

f Ž n. Ž 0 . .

Ž 4.16 .

In this case, the remainder takes the form Rn s

xn n!

f Ž n. Ž ␪n x . .

Ž 4.17 .

The sum of the first n terms in Maclaurin’s series provides an approximation for the value of f Ž x .. The size of the remainder determines how close the sum is to f Ž x .. Since the remainder depends on ␪n , which lies in the interval Ž0, 1., an upper bound on R n that is free of ␪n will therefore be needed to assess the accuracy of the approximation. For example, let us consider the function f Ž x . s cos x. In this case,

ž

f Ž n. Ž x . s cos xq

n␲ 2

/

,

n s 1, 2, . . . ,

and f Ž n. Ž 0 . s cos s

½

ž / n␲ 2

0, nr2 Ž y1. ,

n odd, n even.

112

DIFFERENTIATION

Formula Ž4.16. becomes cos xs 1 q

⬁

Ý

Ž y1.

x2n

n

Ž 2 n. !

ns1

s1y

x2

q

2!

x4 4!

y ⭈⭈⭈ q Ž y1 .

x2n

n

Ž 2 n. !

q R 2 nq1 ,

where from formula Ž4.17. R 2 nq1 is R 2 nq1 s

x 2 nq1

Ž 2 n q 1. !

Ž 2 n q 1. ␲

cos ␪ 2 nq1 xq

2

.

An upper bound on < R 2 nq1 < is then given by < R 2 nq1 < F

< x < 2 nq1

Ž 2 n q 1. !

.

Therefore, the error of approximating cos x with the sum s2 n s 1 y

x2

q

2!

x4 4!

y ⭈⭈⭈ q Ž y1 .

n

x2n

Ž 2 n. !

does not exceed < x < 2 nq1 rŽ2 n q 1.!, where x is measured in radians. For example, if xs ␲r3 and n s 3, the sum s6 s 1 y

x2

q

2

x4 4!

y

x6 6!

s 0.49996

approximates cosŽ␲r3. with an error not exceeding < x< 7 7!

s 0.00027.

The true value of cosŽ␲r3. is 0.5.

4.4. MAXIMA AND MINIMA OF A FUNCTION In this section we consider the problem of finding the extreme values of a function y s f Ž x . whose derivative f ⬘Ž x . exists in any open set inside its domain of definition.

113

MAXIMA AND MINIMA OF A FUNCTION

Definition 4.4.1. A function f : D ™ R has a local Žor relative. maximum at a point x 0 g D if there exists a ␦ ) 0 such that f Ž x . F f Ž x 0 . for all xg N␦ Ž x 0 . l D. The function f has a local Žor relative. minimum at x 0 if f Ž x . G f Ž x 0 . for all xg N␦ Ž x 0 . l D. Local maxima and minima are referred to as local optima Žor local extrema.. I Definition 4.4.2. A function f : D ™ R has an absolute maximum Žminimum. over D if there exists a point x* g D such that f Ž x . F f Ž x*.w f Ž x . G f Ž x*.x for all xg D. Absolute maxima and minima are called absolute optima Žor extrema.. I The determination of local optima of f Ž x . can be greatly facilitated if f Ž x . is differentiable. Theorem 4.4.1. Let f Ž x . be differentiable on the open interval Ž a, b .. If f Ž x . has a local maximum, or a local minimum, at a point x 0 in Ž a, b ., then f ⬘Ž x 0 . s 0. Proof. Suppose that f Ž x . has a local maximum at x 0 . Then, f Ž x . F f Ž x 0 . for all x in some neighborhood N␦ Ž x 0 . ; Ž a, b .. It follows that f Ž x . y f Ž x0 . xy x 0

½

F0 G0

if x) x 0 , if x- x 0 ,

Ž 4.18 .

Ž . for all x in N␦ Ž x 0 .. As x™ xq 0 , the ratio in 4.18 will have a nonpositive y limit, and if x™ x 0 , the ratio will have a nonnegative limit. Since f ⬘Ž x 0 . exists, these two limits must be equal and equal to f ⬘Ž x 0 . as x™ x 0 . We therefore conclude that f ⬘Ž x 0 . s 0. The proof when f Ž x . has a local minimum is similar. I It is important here to note that f ⬘Ž x 0 . s 0 is a necessary condition for a differentiable function to have a local optimum at x 0 . It is not, however, a sufficient condition. That is, if f ⬘Ž x 0 . s 0, then it is not necessarily true that x 0 is a point of local optimum. For example, the function f Ž x . s x 3 has a zero derivative at the origin, but f Ž x . does not have a local optimum there Žwhy not?.. In general, a value x 0 for which f ⬘Ž x 0 . s 0 is called a stationary value for the function. Thus a stationary value does not necessarily correspond to a local optimum. We should also note that Theorem 4.4.1 assumes that f Ž x . is differentiable in a neighborhood of x 0 . If this condition is not fulfilled, the theorem ceases to be true. The existence of f ⬘Ž x 0 . is not prerequisite for f Ž x . to have a local optimum at x 0 . In fact, f Ž x . can have a local optimum at x 0 even if f ⬘Ž x 0 . does not exist. For example, f Ž x . s < x < has a local minimum at xs 0, but f ⬘Ž0. does not exist.

114

DIFFERENTIATION

We recall from Corollary 3.4.1 that if f Ž x . is continuous on w a, b x, then it must achieve its absolute optima at some points inside w a, b x. These points can be interior points, that is, points that belong to the open interval Ž a, b ., or they can be end Žboundary. points. In particular, if f ⬘Ž x . exists on Ž a, b ., to determine the locations of the absolute optima we must solve the equation f ⬘Ž x . s 0 and then compare the values of f Ž x . at the roots of this equation with f Ž a. and f Ž b .. The largest Žsmallest. of these values is the absolute maximum Žminimum.. In the event f ⬘Ž x . / 0 on Ž a, b ., then f Ž x . must achieve its absolute optimum at an end point. 4.4.1. A Sufficient Condition for a Local Optimum We shall make use of Taylor’s expansion to come up with a sufficient condition for f Ž x . to have a local optimum at x s x 0 . Suppose that f Ž x . has n derivatives in a neighborhood N␦ Ž x 0 . such that f ⬘Ž x 0 . s f ⬙ Ž x 0 . s ⭈⭈⭈ s f Ž ny1. Ž x 0 . s 0, but f Ž n. Ž x 0 . / 0. Then by Taylor’s theorem we have f Ž x . s f Ž x0 . q

hn n!

f Ž n. Ž x 0 q ␪n h .

for any x in N␦ Ž x 0 ., where h s xy x 0 and 0 - ␪n - 1. Furthermore, if we assume that f Ž n. Ž x . is continuous at x 0 , then f Ž n. Ž x 0 q ␪n h . s f Ž n. Ž x 0 . q o Ž 1 . , where, as we recall from Section 3.3, oŽ1. ™ 0 as h ™ 0. We can therefore write f Ž x . y f Ž x0 . s

hn n!

f Ž n. Ž x 0 . q o Ž h n . .

Ž 4.19 .

In order for f Ž x . to have a local optimum at x 0 , f Ž x . y f Ž x 0 . must have the same sign Žpositive or negative. for small values of h inside a neighborhood of 0. But, from Ž4.19., the sign of f Ž x . y f Ž x 0 . is determined by the sign of h n f Ž n. Ž x 0 .. We can then conclude that if n is even, then a local optimum is achieved at x 0 . In this case, a local maximum occurs at x 0 if f Ž n. Ž x 0 . - 0, whereas f Ž n. Ž x 0 . ) 0 indicates that x 0 is a point of local minimum. If, however, n is odd, then x 0 is not a point of local optimum, since f Ž x . y f Ž x 0 . changes sign around x 0 . In this case, the point on the graph of y s f Ž x . whose abscissa is x 0 is called a saddle point. In particular, if f ⬘Ž x 0 . s 0 and f ⬙ Ž x 0 . / 0, then x 0 is a point of local optimum. When f ⬙ Ž x 0 . - 0, f Ž x . has a local maximum at x 0 , and when f ⬙ Ž x 0 . ) 0, f Ž x . has a local minimum at x 0 .

115

APPLICATIONS IN STATISTICS

EXAMPLE 4.4.1. Let f Ž x . s 2 x 3 y 3 x 2 y 12 xq 6. Then f ⬘Ž x . s 6 x 2 y 6 x y 12 s 0 at xs y1, 2, and f⬙ Ž x. s

½

12 x y 6 s y18 18

at x s y1, at x s 2.

We have then a local maximum at xs y1 and a local minimum at xs 2. EXAMPLE 4.4.2.

f Ž x . s x 4 y 1. In this case, f ⬘ Ž x . s 4 x 3 s 0 at xs 0, f ⬙ Ž x . s 12 x 2 s 0 at xs 0, f ⵮ Ž x . s 24 xs 0 at xs 0, f Ž4. Ž x . s 24.

Then, xs 0 is a point of local minimum. EXAMPLE 4.4.3.

Consider f Ž x . s Ž xq 5. 2 Ž x 3 y 10.. We have f ⬘ Ž x . s 5 Ž xq 5 . Ž x y 1 . Ž xq 2 . , 2

f ⬙ Ž x . s 10 Ž xq 2 . Ž 2 x 2 q 8 x y 1 . , f ⵮ Ž x . s 10 Ž 6 x 2 q 24 xq 15 . . Here, f ⬘Ž x . s 0 at xs y5, y2, and 1. At xs y5 there is a local maximum, since f ⬙ Žy5. s y270 - 0. At xs 1 we have a local minimum, since f ⬙ Ž1. s 270 ) 0. However, at xs y2 a saddle point occurs, since f ⬙ Žy2. s 0 and f ⵮Žy2. s y90 / 0. EXAMPLE 4.4.4.

f Ž x . s Ž2 xq 1.rŽ x q 4., 0 F xF 5. Then f ⬘ Ž x . s 7r Ž xq 4 . . 2

In this case, f ⬘Ž x . does not vanish anywhere in Ž0, 5.. Thus f Ž x . has no local maxima or local minima in that open interval. Being continuous on w0, 5x, f Ž x . must achieve its absolute optima at the end points. Since f ⬘Ž x . ) 0, f Ž x . is strictly monotone increasing on w0, 5x by Corollary 4.2.1. Its absolute minimum and absolute maximum are therefore attained at x s 0 and xs 5, respectively.

4.5. APPLICATIONS IN STATISTICS Differential calculus has many applications in statistics. Let us consider some of these applications.

116

DIFFERENTIATION

4.5.1. Functions of Random Variables Let Y be a continuous random variable whose cumulative distribution function is F Ž y . s P Ž Y F y .. If F Ž y . is differentiable for all y, then its derivative F⬘Ž y . is called the density function of Y and is denoted by f Ž y .. Continuous random variables for which f Ž y . exists are said to be absolutely continuous. Let Y be an absolutely continuous random variable, and let W be another random variable which can be expressed as a function of Y of the form W s ␺ Ž Y .. Suppose that this function is strictly monotone and differentiable over its domain. By Theorem 3.5.1, ␺ has a unique inverse ␺y1 , which is also differentiable by Theorem 4.2.4. Let G Ž w . denote the cumulative distribution function of W. If ␺ is strictly monotone increasing, then G Ž w . s P Ž W F w . s P Y F ␺y1 Ž w . s F ␺y1 Ž w . . If it is strictly monotone decreasing, then G Ž w . s P Ž W F w . s P Y G ␺y1 Ž w . s 1 y F ␺y1 Ž w . . By differentiating G Ž w . using the chain rule we obtain the density function g Ž w . for W, namely, g Ž w. s

dF ␺y1 Ž w .

d ␺y1 Ž w .

d ␺y1 Ž w .

dw

s f ␺y1 Ž w .

d ␺y1 Ž w .

Ž 4.20 .

dw

if ␺ is strictly monotone increasing, and g Ž w. sy

dF ␺y1 Ž w .

d ␺y1 Ž w .

d ␺y1 Ž w .

dw

s yf ␺y1 Ž w .

d ␺y1 Ž w .

Ž 4.21 .

dw

if ␺ is strictly monotone decreasing. By combining Ž4.20. and Ž4.21. we obtain g Ž w . s f ␺y1 Ž w .

d ␺y1 Ž w . dw

.

Ž 4.22 .

For example, suppose that Y has the uniform distribution U Ž0, 1. whose density function is f Ž y. s

½

1, 0

0 - y - 1, elsewhere .

117

APPLICATIONS IN STATISTICS

Let W s ylog Y. Using formula Ž4.22., the density function of W is given by g Ž w. s

½

eyw , 0

0 - w - ⬁, elsewhere .

The Mean and Variance of W s ␺ (Y) The mean and variance of the random variable W can be obtained by using its density function: EŽW . s

⬁

Hy⬁wg Ž w . dw,

Var Ž W . s E W y E Ž W . s

⬁

Hy⬁

2

wyEŽW .

2

g Ž w . dw.

In some cases, however, the exact distribution of Y may not be known, or g Ž w . may be a complicated function to integrate. In such cases, approximate expressions for the mean and variance of W can be obtained by applying Taylor’s expansion around the mean of Y, ␮. If we assume that ␺ ⬙ Ž y . exists, then

␺ Ž y . s ␺ Ž ␮ . q Ž y y ␮ . ␺ ⬘Ž ␮ . q o Ž y y ␮ . . If oŽ y y ␮ . is small enough, first-order approximations of E ŽW . and VarŽW . can be obtained, namely, EŽW . f␺ Ž ␮. ,

since E Ž Y y ␮ . s 0;

Var Ž W . f ␴ 2 ␺ ⬘ Ž ␮ . , Ž 4.23 . 2

where ␴ 2 s VarŽ Y ., and the symbol f denotes approximate equality. If oŽ y y ␮ . is not small enough, then higher-order approximations can be utilized provided that certain derivatives of ␺ Ž y . exist. For example, if ␺ ⵮Ž y . exists, then

␺ Ž y . s ␺ Ž ␮ . q Ž y y ␮ . ␺ ⬘ Ž ␮ . q 12 Ž y y ␮ . ␺ ⬙ Ž ␮ . q o Ž y y ␮ . . 2

2

In this case, if owŽ y y ␮ . 2 x is small enough, then second-order approximations can be obtained for E ŽW . and VarŽW . of the form E Ž W . f ␺ Ž ␮ . q 12 ␴ 2␺ ⬙ Ž ␮ . , Var Ž W . f E
Q Ž Y . y E Q Ž Y .

since E Ž Y y ␮ . s ␴ 2 , 2

42 ,

118

DIFFERENTIATION

where Q Ž Y . s ␺ Ž ␮ . q Ž Y y ␮ . ␺ ⬘ Ž ␮ . q 12 Ž Y y ␮ . ␺ ⬙ Ž ␮ . . 2

Thus, Var Ž W . f ␴ 2 ␺ ⬘ Ž ␮ .

2

q 14 ␺ ⬙ Ž ␮ .

2

Var Ž Y y ␮ .

2

q ␺ ⬘Ž ␮ . ␺ ⬙ Ž ␮ . E Ž Y y ␮ . . 3

Variance Stabilizing Transformations One of the basic assumptions of regression and analysis of variance is the constancy of the variance ␴ 2 of a response variable Y on which experimental data are obtained. This assumption is often referred to as the assumption of homoscedasticity. There are situations, however, in which ␴ 2 is not constant for all the data. When this happens, Y is said to be heteroscedastic. Heteroscedasticity can cause problems and difficulties in connection with the statistical analysis of the data Žfor a survey of the problems of heteroscedasticity, see Judge et al., 1980.. Some situations that lead to heteroscedasticity are Žsee Wetherill et al., 1986, page 200.: i. The Use of A®eraged Data. In many experimental situations, the data used in a regression program consist of averages of samples that are different in size. This happens sometimes in survey analysis. ii. Variances Depending on the Explanatory Variables. The variance of an observation can sometimes depend on the explanatory Žor input. variables in the hypothesized model, as is the case with some econometric models. For example, if the response variable is household expenditure and one explanatory variable is household income, then the variance of the observations may be a function of household income. iii. Variances Depending on the Mean Response. The response variable Y may have a distribution whose variance is a function of its mean, that is, ␴ 2 s hŽ ␮ ., where ␮ is the mean of Y. The Poisson distribution, for example, has the property that ␴ 2 s ␮. Thus as ␮ changes Žas a function of some explanatory variables., then so will ␴ 2 . The following example illustrates this situation Žsee Chatterjee and Price, 1977, page 39.: Let Y be the number of accidents, and x be the speed of operating a lathe in a machine shop. Suppose that a linear relationship is assumed between Y and x of the form Y s ␤ 0 q ␤ 1 xq ⑀ , where ⑀ is a random error with a zero mean. Here, Y has the Poisson distribution with mean ␮ s ␤ 0 q ␤ 1 x. The variance of Y, being equal to ␮ , will not be constant, since it depends on x.

119

APPLICATIONS IN STATISTICS

Heteroscedasticity due to dependence on the mean response can be removed, or at least reduced, by a suitable transformation of the response variable Y. So let us suppose that ␴ 2 s hŽ ␮ .. Let W s ␺ Ž Y .. We need to find a proper transformation ␺ that causes W to have almost the constant variance property. If this can be accomplished, then ␺ is referred to as a variance stabilizing transformation. If the first-order approximation of VarŽW . by Taylor’s expansion is adequate, then by formula Ž4.23. we can select ␺ so that hŽ ␮ . ␺ ⬘Ž ␮ .

2

s c,

Ž 4.24 .

where c is a constant. Without loss of generality, let c s 1. A solution of Ž4.24. is given by

␺ Ž ␮. s

d␮

H 'h Ž ␮ .

.

Thus if W s ␺ Ž Y ., then VarŽW . will have a variance approximately equal to one. For example, if hŽ ␮ . s ␮ , as is the case with the Poisson distribution, then

␺ Ž ␮. s

d␮

H '␮ s 2'␮ .

Hence, W s 2'Y will have a variance approximately equal to one. ŽIn this case, it is more common to use the transformation W s 'Y which has a variance approximately equal to 0.25.. Thus in the earlier example regarding the relationship between the number of accidents and the speed of operating a lathe, we need to regress 'Y against x in order to ensure approximate homosecdasticity. The relationship Žif any. between ␴ 2 and ␮ may be determined by theoretical considerations based on a knowledge of the type of data usedᎏfor example, Poisson data. In practice, however, such knowledge may not be known a priori. In this case, the appropriate transformation is selected empirically on the basis of residual analysis of the data. See, for example, Box and Draper Ž1987, Chapter 8., Montgomery and Peck Ž1982, Chapter 3.. If possible, a transformation is selected to correct nonnormality Žif the original data are not believed to be normally distributed . as well as heteroscedasticity. In this respect, a useful family of transformations introduced by Box and Cox Ž1964. can be used. These authors considered the power family of transformations defined by

␺ ŽY . s

½

Ž Y ␭ y 1 . r␭ , log Y ,

␭ / 0, ␭ s 0.

120

DIFFERENTIATION

This family may only be applied when Y has positive values. Furthermore, since by l’Hospital’s rule

lim

␭™0

Y ␭y1

␭

s lim Y ␭ log Y s log Y , ␭™0

the Box᎐Cox transformation is a continuous function of ␭. An estimate of ␭ can be obtained from the data using the method of maximum likelihood Žsee Montgomery and Peck, 1982, Section 3.7.1; Box and Draper, 1987, Section 8.4.. Asymptotic Distributions The asymptotic distributions of functions of random variables are of special interest in statistical limit theory. By definition, a sequence of random variables
Yn4⬁ns1 converges in distribution to the random variable Y if lim Fn Ž y . s F Ž y .

n™⬁

at each point y where F Ž y . is continuous, where FnŽ y . is the cumulative distribution function of Yn Ž n s 1, 2, . . . . and F Ž y . is the cumulative distribution function of Y Žsee Section 5.3 concerning sequences of functions.. This form of convergence is denoted by writing d

Yn ™ Y . An illustration of convergence in distribution is provided by the well-known central limit theorem. It states that if
Yn4⬁ns1 is a sequence of independent and identically distributed random variables with common mean and variance, ␮ and ␴ 2 , respectively, that are both finite, and if Yn s Ý nis1 Yirn is the sample mean of a sample size n, then as n ™ ⬁, Yn y ␮

␴r'n

d

™ Z,

where Z has the standard normal distribution N Ž0, 1.. An extension of the central limit theorem that includes functions of random variables is given by the following theorem: Theorem 4.5.1. Let
Yn4⬁ns1 be a sequence of independent and identically distributed random variables with mean ␮ and variance ␴ 2 Žboth finite., and let Yn be the sample mean of a sample of size n. If ␺ Ž y . is a function whose derivative ␺ ⬘Ž y . exists and is continuous in a neighborhood of ␮ such that

121

APPLICATIONS IN STATISTICS

␺ ⬘Ž ␮ . / 0, then as n ™ ⬁, ␺ Ž Yn . y ␺ Ž ␮ . ␴ ␺ ⬘ Ž ␮ . r'n

Proof. See Wilks Ž1962, page 259..

d

™ Z. I

On the basis of Theorem 4.5.1 we can assert that when n is large enough, ␺ Ž Yn . is approximately distributed as a normal variate with a mean ␺ Ž ␮ . and a standard deviation Ž ␴r 'n .< ␺ ⬘Ž ␮ .< . For example, if ␺ Ž y . s y 2 , then as n ™ ⬁, Yn2 y ␮ 2

2 < ␮ < ␴r'n

d

™ Z.

4.5.2. Approximating Response Functions Perhaps the most prevalent use of Taylor’s expansion in statistics is in the area of linear models. Let Y denote a response variable, such as the yield of a product, whose mean ␮ Ž x . is believed to depend on an explanatory Žor input. variable x such as temperature or pressure. The true relationship between ␮ and x is usually unknown. However, if ␮ Ž x . is considered to have derivatives of all orders, then it is possible to approximate its values by using low-order terms of a Taylor’s series over a limited range of interest. In this case, ␮ Ž x . can be represented approximately by a polynomial of degree d ŽG 1. of the form

␮ Ž x . s ␤0 q

d

Ý ␤j x j ,

js1

where ␤ 0 , ␤ 1 , . . . , ␤ d are unknown parameters. Estimates of these parameters are obtained by running n ŽG dq 1. experiments in which n observations, y 1 , y 2 , . . . , yn , on Y are obtained for specified values of x. This leads us to the linear model d

yi s ␤ 0 q

Ý ␤ j x ij q ⑀ i ,

i s 1, 2, . . . , n,

Ž 4.25 .

js1

where ⑀ i is a random error. The method of least squares can then be used to estimate the unknown parameters in Ž4.25.. The adequacy of model Ž4.25. to represent the true mean response ␮ Ž x . can be checked using the given data provided that replicated observations are available at some points inside the region of interest. For more details concerning the adequacy of fit of linear models and the method of least squares, see, for example, Box and Draper Ž1987, Chapters 2 and 3. and Khuri and Cornell Ž1996, Chapter 2..

122

DIFFERENTIATION

4.5.3. The Poisson Process A random phenomenon that arises through a process which continues in time Žor space. in a manner controlled by the laws of probability is called a stochastic process. A particular example of such a process is the Poisson process, which is associated with the number of events that take place over a period of timeᎏfor example, the arrival of customers at a service counter, or the arrival of ␣-rays, emitted from a radioactive source, at a Geiger counter. Define pnŽ t . as the probability of n arrivals during a time interval of length t. For a Poisson process, the following postulates are assumed to hold: 1. The probability of exactly one arrival during a small time interval of length h is approximately proportional to h, that is, p1 Ž h . s ␭ h q o Ž h . as h ™ 0, where ␭ is a constant. 2. The probability of more than one arrival during a small time interval of length h is negligible, that is,

Ý

pn Ž h . s o Ž h .

n)1

as h ™ 0. 3. The probability of an arrival occurring during a small time interval Ž t, t q h. does not depend on what happened prior to t. This means that the events defined according to the number of arrivals occurring during nonoverlapping time intervals are independent. On the basis of the above postulates, an expression for pnŽ t . can be found as follows: For n G 1 and for small h we have approximately pn Ž t q h . s pn Ž t . p 0 Ž h . q pny1 Ž t . p1 Ž h . s pn Ž t . 1 y ␭ h q o Ž h . q pny1 Ž t . ␭ h q o Ž h . ,

Ž 4.26 .

since the probability of no arrivals during the time interval Ž t, t q h. is approximately equal to 1 y p1Ž h.. For n s 0 we have p0 Ž t q h . s p0 Ž t . p0 Ž h . s p0 Ž t . 1 y ␭ h q o Ž h . .

Ž 4.27 .

123

APPLICATIONS IN STATISTICS

From Ž4.26. and Ž4.27. we then get for n G 1, pn Ž t q h . y pn Ž t . h

s pn Ž t . y␭ q

oŽ h. h

q pny1 Ž t . ␭ q

oŽ h. h

,

and for n s 0, p0 Ž t q h . y p0 Ž t . h

s p 0 Ž t . y␭ q

oŽ h. h

.

By taking the limit as h ™ 0 we obtain the derivatives pXn Ž t . s y␭ pn Ž t . q ␭ pny1 Ž t . ,

nG1

pX0 Ž t . s y␭ p 0 Ž t . .

Ž 4.28 . Ž 4.29 .

From Ž4.29. the solution for p 0 Ž t . is given by p 0 Ž t . s ey␭ t ,

Ž 4.30 .

since p 0 Ž t . s 1 when t s 0 Žthat is, initially there were no arrivals.. By substituting Ž4.30. in Ž4.28. when n s 1 we get pX1 Ž t . s y␭ p1 Ž t . q ␭ ey␭ t .

Ž 4.31 .

If we now multiply the two sides of Ž4.31. by e ␭t we obtain e ␭t pX1 Ž t . q ␭ p1 Ž t . e ␭t s ␭ , or e ␭t p1 Ž t . ⬘ s ␭ . Hence, e ␭t p1 Ž t . s ␭ t q c, where c is a constant. This constant must be equal to zero, since p1Ž0. s 0. We then have p1 Ž t . s ␭ tey␭ t . By continuing in this process and using equation Ž4.28. we can find p 2 Ž t ., then p 3 Ž t ., . . . , etc. In general, it can be shown that pn Ž t . s

ey␭ t Ž ␭ t . n!

n

,

n s 0, 1, 2, . . . .

Ž 4.32 .

124

DIFFERENTIATION

In particular, if t s 1, then formula Ž4.32. gives the probability of n arrivals during one unit of time, namely, pn Ž 1 . s

ey␭␭ n n!

n s 0, 1, . . . .

,

This gives the probability mass function of a Poisson random variable with mean ␭. 4.5.4. Minimizing the Sum of Absolute Deviations Consider a data set consisting of n observations y 1 , y 2 , . . . , yn . For an arbitrary real number a, let DŽ a. denote the sum of absolute deviations of the data from a, that is, D Ž a. s

n

Ý < yi y a < .

is1

For a given a, DŽ a. represents a measure of spread, or variation, for the data set. Since the value of DŽ a. varies with a, it may be of interest to determine its minimum. We now show that DŽ a. is minimized when as ␮*, where ␮* denotes the median of the data set. By definition, ␮* is a value that falls in the middle when the observations are arranged in order of magnitude. It is a measure of location like the mean. If we write yŽ1. F yŽ2. F ⭈⭈⭈ F yŽ n. for the ordered yi ’s, then when n is odd, ␮* is the unique value yŽ n r2q1r2. ; whereas when n is even, ␮* is any value such that yŽ n r2. F ␮* F yŽ n r2q1. . In the latter case, ␮* is sometimes chosen as the middle of the interval. There are several ways to show that ␮* minimizes DŽ a.. The following simple proof is due to Blyth Ž1990.: On the interval yŽ k . - a- yŽ kq1. , k s 1, 2, . . . , n y 1, we have D Ž a. s

n

Ý < yŽ i. y a <

is1 k

s

Ý

Ž ay yŽ i. . q

is1

n

Ý Ž yŽ i. y a.

iskq1 k

s kay

Ý

is1

n

yŽ i. q

Ý

yŽ i. y Ž n y k . a.

iskq1

The function DŽ a. is continuous for all a and is differentiable everywhere except at y 1 , y 2 , . . . , yn . For a/ yi Ž i s 1, 2, . . . , n., the derivative D⬘Ž a. is given by D⬘ Ž a . s 2 Ž k y nr2. .

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

125

If k / nr2, then D⬘Ž a. / 0 on Ž yŽ k . , yŽ kq1. ., and by Corollary 4.2.1, DŽ a. must be strictly monotone on w yŽ k . , yŽ kq1. x, k s 1, 2, . . . , n y 1. Now, when n is odd, DŽ a. is strictly monotone decreasing for aF yŽ n r2q1r2. , because D⬘Ž a. - 0 over Ž yŽ k . , yŽ kq1. . for k - nr2. It is strictly monotone increasing for aG yŽ n r2q1r2. , because D⬘Ž a. ) 0 over Ž yŽ k . , yŽ kq1. . for k ) nr2. Hence, ␮* s yŽ n r2q1r2. is a point of absolute minimum for DŽ a.. Furthermore, when n is even, DŽ a. is strictly monotone decreasing for aF yŽ n r2. , because D⬘Ž a. - 0 over Ž yŽ k . , yŽ kq1. . for k - nr2. Also, DŽ a. is constant over Ž yŽ n r2. , yŽ n r2q1. ., because D⬘Ž a. s 0 for k s nr2, and is strictly monotone increasing for aG yŽ n r2q1. , because D⬘Ž a. ) 0 over Ž yŽ k . , yŽ kq1. . for k ) nr2. This indicates that DŽ a. achieves its absolute minimum at any point ␮* such that yŽ n r2. F ␮* F yŽ n r2q1. , which completes the proof.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Apostol, T. M. Ž1964.. Mathematical Analysis. Addison-Wesley, Reading, Massachusetts. ŽChap. 5 discusses differentiation of functions of one variable.. Blyth, C. R. Ž1990.. ‘‘Minimizing the sum of absolute deviations.’’ Amer. Statist., 44, 329. Box, G. E. P., and D. R. Cox Ž1964.. ‘‘An analysis of transformations.’’ J. Roy. Statist. Soc. Ser. B, 26, 211᎐243. Box, G. E. P., and N. R. Draper Ž1987.. Empirical Model-Building and Response Surfaces. Wiley, New York. ŽChap. 2 introduces the idea of approximating the mean of a response variable using low-order polynomials; Chap. 3 discusses the method of least squares for fitting empirical models; the use of transformations, including those for stabilizing variances, is described in Chap. 8.. Box, G. E. P., W. G. Hunter, and J. S. Hunter Ž1978.. Statistics for Experimenters. Wiley, New York. ŽVarious transformations are listed in Chap. 7, which include the Box᎐Cox and variance stabilizing transformations. . Buck, R. C. Ž1956.. Ad®anced Calculus, McGraw-Hill, New York. ŽChap. 2 discusses the mean value theorem and l’Hospital’s rule.. Chatterjee, S., and B. Price Ž1977.. Regression Analysis by Example. Wiley, New York. ŽChap. 2 includes a discussion concerning variance stabilizing transformations, in addition to detection and removal of the effects of heteroscedasticity in regression analysis. . Cooke, W. P. Ž1988.. ‘‘L’Hopital’s rule in a Poisson derivation.’’ Amer. Math. Monthly, ˆ 95, 253᎐254. Eggermont, P. P. B. Ž1988.. ‘‘Noncentral difference quotients and the derivative.’’ Amer. Math. Monthly, 95, 551᎐553. Eves, H. Ž1976.. An Introduction to the History of Mathematics, 4th ed. Holt, Rinehart and Winston, New York. Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. ŽDifferentiation is the subject of Chap. 4..

126

DIFFERENTIATION

Georgiev, A. A. Ž1984.. ‘‘Kernel estimates of functions and their derivatives with applications,’’ Statist. Probab. Lett., 2, 45᎐50. Hardy, G. H. Ž1955.. A Course of Pure Mathematics, 10th ed. The University Press, Cambridge, England. ŽChap. 6 covers differentiation and provides some interesting examples. . Hogg, R. V., and A. T. Craig Ž1965.. Introduction to Mathematical Statistics, 2nd ed. Macmillan, New York. ŽChap. 4 discusses distributions of functions of random variables. . James, A. T., and R. A. J. Conyers Ž1985.. ‘‘Estimation of a derivative by a difference quotient: Its application to hepatocyte lactate metabolism.’’ Biometrics, 41, 467᎐476. Judge, G. G., W. E. Griffiths, R. C. Hill, and T. C. Lee Ž1980.. The Theory and Practice of Econometrics. Wiley, New York. Khuri, A. I., and J. A. Cornell Ž1996.. Response Surfaces, 2nd ed. Dekker, New York. ŽChaps. 1 and 2 discuss the polynomial representation of a response surface and the method of least squares. . Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽSection 3.2 discusses the development of the Poisson process. . Menon, V. V., B. Prasad, and R. S. Singh Ž1984.. ‘‘Non-parametric recursive estimates of a probability density function and its derivatives.’’ J. Statist. Plann. Inference, 9, 73᎐82. Montgomery, D. C., and E. A. Peck Ž1982.. Introduction to Linear Regression Analysis. Wiley, New York. ŽChap. 3 presents several methods useful for checking the validity of the basic regression assumptions. Several variance stabilizing transformations are also listed. . Parzen, E. Ž1962.. Stochastic Processes. Holden-Day, San Francisco. ŽChap. 1 introduces the definition of stochastic processes including the Poisson process.. Roberts, A. W., and D. E. Varberg Ž1973.. Con®ex Functions. Academic Press, New York. ŽChap. 1 discusses a characterization of convex functions using derivatives; Chap. 5 discusses maxima and minima of differentiable functions. . Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts. ŽChap. 3 discusses absolutely continuous random variables. . Rudin, W. Ž1964.. Principles of Mathematical Analysis. 2nd ed. McGraw-Hill, New York. ŽDifferentiation is discussed in Chap. 5.. Sagan, H. Ž1974.. Ad®anced Calculus. Houghton Mifflin, Boston. ŽChap. 3 discusses differentiation. . Wetherill, G. B., P. Duncombe, M. Kenward, J. Kollerstrom, ¨ ¨ S. R. Paul, and B. J. Vowden Ž1986.. Regression Analysis with Applications. Chapman and Hall, London, England. ŽSection 9.2 discusses the sources of heteroscedasticity in regression analysis. . Wilks, S. S. Ž1962.. Mathematical Statistics. Wiley, New York. ŽChap. 9 considers limit theorems including asymptotic distributions of functions of the sample mean..

127

EXERCISES

EXERCISES In Mathematics 4.1. Let f Ž x . be defined in a neighborhood of the origin. Show that if f ⬘Ž0. exists, then lim

f Ž h . y f Ž yh . 2h

h™0

s f ⬘ Ž 0. .

Give a counterexample to show that the converse is not true in general, that is, if the above limit exists, then it is not necessary that f ⬘Ž0. exists. 4.2. Let f Ž x . and g Ž x . have derivatives up to order n on w a, b x. Let hŽ x . s f Ž x . g Ž x .. Show that hŽ n. Ž x . s

n

Ý

ks0

ž/

n Žk. f Ž x . g Ž nyk . Ž x . . k

ŽThis is known as Leibniz’s formula.. 4.3. Suppose that f Ž x . has a derivative at a point x 0 , a- x 0 - b. Show that there exists a neighborhood N␦ Ž x 0 . and a positive number A such that f Ž x . y f Ž x 0 . - A < xy x 0 < for all xg N␦ Ž x 0 ., x/ x 0 . 4.4. Suppose that f Ž x . is differentiable on Ž0, ⬁. and f ⬘Ž x . ™ 0 as x™ ⬁. Let g Ž x . s f Ž xq 1. y f Ž x .. Prove that g Ž x . ™ 0 as x ™ ⬁. 4.5. Let the function f Ž x . be defined as f Ž x. s

½

x 3 y 2 x, ax 2 y bxq 1,

xG 1, x- 1.

For what values of a and b does f Ž x . have a continuous derivative? 4.6. Suppose that f Ž x . is twice differentiable on Ž0, ⬁.. Let m 0 , m1 , m 2 be the least upper bounds of < f Ž x .< , < f ⬘Ž x .< , and < f ⬙ Ž x .< , respectively, on Ž0, ⬁..

128

DIFFERENTIATION

(a) Show that f ⬘Ž x . F

m0 h

q hm 2

for all x in Ž0, ⬁. and for every h ) 0. (b) Deduce from Ža. that m12 F 4 m 0 m 2 . 4.7. Suppose that lim x ™ x 0 f ⬘Ž x . exists. Does it follow that f Ž x . is differentiable at x 0? Give a proof to show that the statement is correct or produce a counterexample to show that it is false. 4.8. Show that DŽ a. s Ý nis1 < yi y a < has no derivatives with respect to a at y 1 , y 2 , . . . , yn . 4.9. Suppose that the function f Ž x . is such that f ⬘Ž x . and f ⬙ Ž x . are continuous in a neighborhood of the origin and satisfies f Ž0. s 0. Show that lim x™0

d

f Ž x.

dx

x

s

1 2

f ⬙ Ž 0. .

4.10. Show that if f ⬘Ž x . exists and is bounded for all x, then f Ž x . is uniformly continuous on R, the set of real numbers. 4.11. Suppose that g: R ™ R and that < g ⬘Ž x .< - M for all x g R, where M is a positive constant. Define f Ž x . s xq cg Ž x ., where c is a positive constant. Show that it is possible to choose c small enough so that f is a one-to-one function. 4.12. Suppose that f Ž x . is continuous on w0, ⬁., f ⬘Ž x . exists on Ž0, ⬁., f Ž0. s 0, and f ⬘Ž x . is monotone increasing on Ž0, ⬁.. Show that g Ž x . is monotone increasing on Ž0, ⬁. where g Ž x . s f Ž x .rx. 4.13. Show that if a) 1 and m ) 0, then (a) lim x ™⬁Ž a xrx m . s ⬁, (b) lim x ™⬁wŽlog x .rx m x s 0. 4.14. Apply l’Hospital’s rule to find the limit

ž

lim 1 q

x™⬁

1 x

/

x

.

129

EXERCISES

4.15. (a) Find lim x ™ 0q Žsin x . x. (b) Find lim x ™ 0q Ž ey1r xrx .. 4.16. Show that lim 1 q axq o Ž x .

1rx

s ea,

x™0

where a is a constant and oŽ x . is any function whose order of magnitude is less than that of x as x™ 0. 4.17. Consider the functions f Ž x . s 4 x 3 q 6 x 2 y 10 xq 2 and g Ž x . s 3 x 4 q 4 x 3 y 5 x 2 q 1. Show that f ⬘Ž x . g ⬘Ž x .

/

f Ž 1. y f Ž 0. g Ž 1. y g Ž 0.

for any xg Ž0, 1.. Does this contradict Cauchy’s mean value theorem? 4.18. Suppose that f Ž x . is differentiable for aF xF b. If f ⬘Ž a. - f ⬘Ž b . and ␥ is a number such that f ⬘Ž a. - ␥ - f ⬘Ž b ., show that there exists a ␰ , a- ␰ - b, for which f ⬘Ž ␰ . s ␥ wa similar result holds if f ⬘Ž a. ) f ⬘Ž b .x. w Hint: Consider the function g Ž x . s f Ž x . y ␥ Ž xy a.. Show that g Ž x . has a minimum at ␰ .x 4.19. Suppose that f Ž x . is differentiable on Ž a, b .. Let x 1 , x 2 , . . . , x n be in Ž a,b ., and let ␭1 , ␭2 , . . . , ␭ n be positive numbers such that Ý nis1 ␭ i s 1. Show that there exists a point c in Ž a, b . such that n

Ý ␭i f ⬘ Ž x i . s f ⬘ Ž c . .

is1

w Note: This is a generalization of the result in Exercise 4.18.x 4.20. Let x 1 , x 2 , . . . , x n and y 1 , y 2 , . . . , yn be in Ž a, b . such that x i - yi Ž i s 1, 2, . . . , n.. Show that if f Ž x . is differentiable on Ž a, b ., then there exists a point c in Ž a, b . such that n

Ý

is1

f Ž yi . y f Ž x i . s f ⬘ Ž c .

n

Ý Ž yi y x i . .

is1

4.21. Give a Maclaurin’s series expansion of the function f Ž x . s logŽ1 q x .. 4.22. Discuss the maxima and minima of the function f Ž x . s Ž x 4 q 3.rŽ x 2 q 2..

130

DIFFERENTIATION

4.23. Determine if f Ž x . s e 3 xrx has an absolute minimum on Ž0, ⬁.. 4.24. For what values of a and b is the function f Ž x. s

1 x q axq b 2

bounded on the interval wy1, 1x? Find the absolute maximum on that interval. In Statistics 4.25. Let Y be a continuous random variable whose cumulative distribution function, F Ž y ., is strictly monotone. Let G Ž y . be another strictly monotone, continuous cumulative distribution function. Show that the cumulative distribution function of the random variable Gy1 w F Ž Y .x is G Ž y .. 4.26. Let Y have the cumulative distribution function FŽ y. s

½

1 y eyy , 0,

y G 0, y - 0.

Find the density function of W s 'Y . 4.27. Let Y be normally distributed with mean 1 and variance 0.04. Let W s Y 3. (a) Find the density function of W. (b) Find the exact mean and variance of W. (c) Find approximate values for the mean and variance of W using Taylor’s expansion, and compare the results with those of Žb.. 4.28. Let Z be normally distributed with mean 0 and variance 1. Let Y s Z 2 . Find the density function of Y. w Note: The function ␺ Ž z . s z 2 is not strictly monotone for all z.x 4.29. Let X be a random variable that denotes the age at failure of a component. The failure rate is defined as the probability of failure in a finite interval of time, say of length h, given the age of the component, say x. This failure rate is therefore equal to P Ž xF X F xq h < X G x . . Consider the following limit: lim h™0

1 h

P Ž xF X F xq h < X G x . .

131

EXERCISES

If this limit exists, then it is called the hazard rate, or instantaneous failure rate. (a) Give an expression for the failure rate in terms of F Ž x ., the cumulative distribution function of X. (b) Suppose that X has the exponential distribution with the cumulative distribution function F Ž x . s 1 y eyx r ␴ ,

xG 0,

where ␴ is a positive constant. Show that X has a constant hazard rate. (c) Show that any random variable with a constant hazard rate must have the exponential distribution. 4.30. Consider a Poisson process with parameter ␭ over the interval Ž0, t .. Divide this interval into n equal subintervals of length h s trn. We consider that we have a ‘‘success’’ in a given subinterval if one arrival occurs in that subinterval. If there are no arrivals, then we consider that we have a ‘‘failure.’’ Let Yn denote the number of ‘‘successes’’ in the n subintervals of length h. Then we have approximately P Ž Yn s r . s

ž/

nyr n r pn Ž 1 y pn . , r

r s 0, 1, . . . , n,

where pn is approximately equal to ␭ h s ␭ trn. Show that lim P Ž Yn s r . s

n™⬁

ey␭ t Ž ␭ t . r!

r

.

CHAPTER 5

Infinite Sequences and Series

The study of the theory of infinite sequences and series is an integral part of advanced calculus. All limiting processes, such as differentiation and integration, can be investigated on the basis of this theory. The first example of an infinite series is attributed to Archimedes, who showed that the sum 1q

1 4

q ⭈⭈⭈ q

1 4n

was less than 43 for any value of n. However, it was not until the nineteenth century that the theory of infinite series was firmly established by AugustinLouis Cauchy Ž1789᎐1857.. In this chapter we shall study the theory of infinite sequences and series, and investigate their convergence. Unless otherwise stated, the terms of all sequences and series considered in this chapter are real-valued.

5.1. INFINITE SEQUENCES In Chapter 1 we introduced the general concept of a function. An infinite sequence is a particular function f : Jq™ R defined on the set of all positive integers. For a given n g Jq, the value of this function, namely f Ž n., is called the nth term of the infinite sequence and is denoted by a n . The sequence itself is denoted by the symbol
a n4⬁ns1 . In some cases, the integer with which the infinite sequence begins is different from one. For example, it may be equal to zero or to some other integer. For the sake of simplicity, an infinite sequence will be referred to as just a sequence. Since a sequence is a function, then, in particular, the sequence
a n4⬁ns1 can have the following properties: 1. It is bounded if there exists a constant K ) 0 such that < a n < F K for all n. 132

133

INFINITE SEQUENCES

2. It is monotone increasing if a n F a nq1 for all n, and is monotone decreasing if a n G a nq1 for all n. 3. It converges to a finite number c if lim n™⬁ a n s c, that is, for a given ⑀ ) 0 there exists an integer N such that < an y c < - ⑀

if n ) N.

In this case, c is called the limit of the sequence and this fact is denoted by writing a n ™ c as n ™ ⬁. If the sequence does not converge to a finite limit, then it is said to be divergent. 4. It is said to oscillate if it does not converge to a finite limit, nor to q⬁ or y⬁ as n ™ ⬁. EXAMPLE 5.1.1. Let a n s Ž n2 q 2 n.rŽ2 n2 q 3.. Then a n ™ 12 as n ™ ⬁, since 1 q 2rn lim a n s lim 2 n™⬁ n™⬁ 2 q 3rn s

1 2

.

EXAMPLE 5.1.2. Consider a n s 'n q 1 y 'n . This sequence converges to zero, since an s s

Ž 'n q 1 y 'n .Ž 'n q 1 q 'n . 'n q 1 q 'n 1

'n q 1 q 'n

.

Hence, a n ™ 0 as n ™ ⬁. EXAMPLE 5.1.3. Suppose that a n s 2 nrn 3. Here, the sequence is divergent, since by Example 4.2.3, 2n lim 3 s ⬁. n™⬁ n EXAMPLE 5.1.4. Let a n s Žy1. n. This sequence oscillates, since it is equal to 1 when n is even and to y1 when n is odd. Theorem 5.1.1.

Every convergent sequence is bounded.

Proof. Suppose that
a n4⬁ns1 converges to c. Then, there exists an integer N such that < an y c < - 1

if n ) N.

134

INFINITE SEQUENCES AND SERIES

For such values of n, we have < a n < - max Ž < c y 1 < , < c q 1 . . It follows that < an < - K for all n, where K s max Ž < a1 < q 1, < a2 < q 1, . . . , < a N < q 1, < c y 1 < , < c q 1 < . .

I

The converse of Theorem 5.1.1 is not necessarily true. That is, if a sequence is bounded, then it does not have to be convergent. As a counterexample, consider the sequence given in Example 5.1.4. This sequence is bounded, but is not convergent. To guarantee converge of a bounded sequence we obviously need an additional condition. Theorem 5.1.2. Every bounded monotone Žincreasing or decreasing . sequence converges. Proof. Suppose that
a n4⬁ns1 is a bounded and monotone increasing sequence Žthe proof is similar if the sequence is monotone decreasing .. Since the sequence is bounded, it must be bounded from above and hence has a least upper bound c Žsee Theorem 1.5.1.. Thus a n F c for all n. Furthermore, for any given ⑀ ) 0 there exists an integer N such that c y ⑀ - a N F c; otherwise c y ⑀ would be an upper bound of
a n4⬁ns1 . Now, because the sequence is monotone increasing, c y ⑀ - a N F a Nq1 F a Nq2 F ⭈⭈⭈ F c, that is, c y ⑀ - an F c

for n G N.

We can write c y ⑀ - an - c q ⑀ , or equivalently, < an y c < - ⑀

if n G N.

This indicates that
a n4⬁ns1 converges to c.

I

135

INFINITE SEQUENCES

Using Theorem 5.1.2 it is easy to prove the following corollary. Corollary 5.1.1. 1. If
a n4⬁ns1 is bounded from above and is monotone increasing, then
a n4⬁ns1 converges to c s sup nG 1 a n . 2. If
a n4⬁ns1 is bounded from below and is monotone decreasing, then
a n4⬁ns1 converges to ds inf nG 1 a n . EXAMPLE 5.1.5. Consider the sequence
a n4⬁ns1 , where a1 s '2 and a nq1 s 2 q a n for n G 1. This sequence is bounded, since a n - 2 for all n, as

'

'

can be easily shown using mathematical induction: We have a1 s '2 - 2. If a n - 2, then a nq1 - 2 q '2 - 2. Furthermore, the sequence is monotone increasing, since a n F a nq1 for n s 1, 2, . . . , which can also be shown by mathematical induction. Hence, by Theorem 5.1.2
a n4⬁ns1 must converge. To find its limit, we note that

'

lim a nq1 s lim

n™⬁

n™⬁

'2 q 'a

( '

s 2q

n

lim a n .

n™⬁

If c denotes the limit of a n as n ™ ⬁, then

'

c s 2 q 'c . By solving this equation under the condition c G '2 we find that the only solution is c s 1.831. Definition 5.1.1. Consider the sequence
a n4⬁ns1 . An infinite collection of its terms, picked out in a manner that preserves the original order of the terms of the sequence, is called a subsequence of
a n4⬁ns1 . More formally, any sequence of the form
bn4⬁ns1 , where bn s a k n such that k 1 - k 2 - ⭈⭈⭈ - k n ⭈⭈⭈ is a subsequence of
a n4⬁ns1 . Note that k n G n for n G 1. I Theorem 5.1.3. A sequence
a n4⬁ns1 converges to c if and only if every subsequence of
a n4⬁ns1 converges to c. Proof. The proof is left to the reader.

I

It should be noted that if a sequence diverges, then it does not necessarily follow that every one of its subsequences must diverge. A sequence may fail to converge, yet several of its subsequences converge. For example, the

136

INFINITE SEQUENCES AND SERIES

sequence whose nth term is a n s Žy1. n is divergent, as was seen earlier. However, the two subsequences
bn4⬁ns1 and
c n4⬁ns1 , where bn s a2 n s 1 and c n s a2 ny1 s y1 Ž n s 1, 2, . . . ., are both convergent. We have noted earlier that a bounded sequence may not converge. It is possible, however, that one of its subsequences is convergent. This is shown in the next theorem. Theorem 5.1.4.

Every bounded sequence has a convergent subsequence.

Proof. Suppose that
a n4⬁ns1 is a bounded sequence. Without loss of generality we can consider that the number of distinct terms of the sequence is infinite. ŽIf this is not the case, then there exists an infinite subsequence of
a n4⬁ns1 that consists of terms that are equal. Obviously, such a subsequence converges.. Let G denote the set consisting of all terms of the sequence. Then G is a bounded infinite set. By Theorem 1.6.2, G must have a limit point, say c. Also, by Theorem 1.6.1, every neighborhood of c must contain infinitely many points of G. It follows that we can find integers k 1 - k 2 - k 3 - ⭈⭈⭈ such that ak n y c -

1 n

for n s 1, 2, . . . .

Thus for a given ⑀ ) 0 there exists an integer N ) 1r⑀ such that < a k n y c < - ⑀ if n ) N. This indicates that the subsequence
a k n 4⬁ns1 converges to c. I We conclude from Theorem 5.1.4 that a bounded sequence can have several convergent subsequences. The limit of each of these subsequences is called a subsequential limit. Let E denote the set of all subsequential limits of
a n4⬁ns1 . This set is bounded, since the sequence is bounded Žwhy?.. Definition 5.1.2. Let
a n4⬁ns1 be a bounded sequence, and let E be the set of all its subsequential limits. Then the least upper bound of E is called the upper limit of
a n4⬁ns1 and is denoted by lim sup n™⬁ a n . Similarly, the greatest lower bound of E is called the lower limit of
a n4⬁ns1 and is denoted by lim inf n™⬁ a n . For example, the sequence
a n4⬁ns1 , where a n s Žy1. nw1 q Ž1rn.x, has two subsequential limits, namely y1 and 1. Thus E s
y1, 14 , and lim sup n™⬁ a n s 1, lim inf n™⬁ a n s y1. I Theorem 5.1.5.

The sequence
a n4⬁ns1 converges to c if any only if lim inf a n s lim sup a n s c. n™⬁

n™⬁

Proof. The proof is left to the reader.

I

137

INFINITE SEQUENCES

Theorem 5.1.5 implies that when a sequence converges, the set of all its subsequential limits consists of a single element, namely the limit of the sequence. 5.1.1. The Cauchy Criterion We have seen earlier that the definition of convergence of a sequence
a n4⬁ns1 requires finding the limit of a n as n ™ ⬁. In some cases, such a limit may be difficult to figure out. For example, consider the sequence whose nth term is an s 1 y

1 3

q

1 5

y

1 7

q ⭈⭈⭈ q

Ž y1.

ny1

2ny1

,

n s 1, 2, . . . .

Ž 5.1 .

It is not easy to calculate the limit of a n in order to find out if the sequence converges. Fortunately, however, there is another convergence criterion for sequences, known as the Cauchy criterion after Augustin-Louis Cauchy Žit was known earlier to Bernhard Bolzano, 1781᎐1848, a Czechoslovakian priest whose mathematical work was undeservedly overlooked by his lay and clerical contemporaries; see Boyer, 1968, page 566.. Theorem 5.1.6 ŽThe Cauchy Criterion.. The sequence
a n4⬁ns1 converges if and only if it satisfies the following condition, known as the ⑀-condition: For each ⑀ ) 0 there is an integer N such that < am y an < - ⑀

for all m ) N, n ) N.

Proof. Necessity: If the sequence converges, then it must satisfy the ⑀-condition. Let ⑀ ) 0 be given. Since the sequence
a n4⬁ns1 converges, then there exists a number c and an integer N such that < an y c < -

⑀ 2

if n ) N.

Hence, for m ) N, n ) N we must have < am y an < s < am y c q c y an < F < am y c < q < an y c < - ⑀ . Sufficiency: If the sequence satisfies the ⑀-condition, then it must converge. If the ⑀-condition is satisfied, then there is an integer N such that for any given ⑀ ) 0, < a n y a Nq1 < - ⑀ for all values of n G N q 1. Thus for such values of n, a Nq1 y ⑀ - a n - a Nq1 q ⑀ .

Ž 5.2 .

138

INFINITE SEQUENCES AND SERIES

The sequence
a n4⬁ns1 is therefore bounded, since from the double inequality Ž5.2. we can assert that < a n < - max Ž < a1 < q 1, < a2 < q 1, . . . , < a N < q 1, < a Nq1 y ⑀ < , < a Nq1 q ⑀ < . for all n. By Theorem 5.1.4,
a n4⬁ns1 has a convergent subsequence
a k n 4⬁ns1 . Let c be the limit of this subsequence. If we invoke again the ⑀-condition, we can find an integer N⬘ such that < am y ak < - ⑀ ⬘ n

if m ) N⬘, k n G n G N⬘,

where ⑀ ⬘ - ⑀ . By fixing m and letting k n ™ ⬁ we get < am y c < F ⑀ ⬘ - ⑀

if m ) N⬘.

This indicates that the sequence
a n4⬁ns1 is convergent and has c as its limit. I Definition 5.1.3. A sequence
a n4⬁ns1 that satisfies the ⑀-condition of the Cauchy criterion is said to be a Cauchy sequence. I EXAMPLE 5.1.6. With the help of the Cauchy criterion it is now possible to show that the sequence
a n4⬁ns1 whose nth term is defined by formula Ž5.1. is a Cauchy sequence and is therefore convergent. To do so, let m ) n. Then, a m y a n s Ž y1 .

1

n

y

2nq1

1 2nq3

q ⭈⭈⭈ q

Ž y1.

py1

,

2 n q 2 py 1

Ž 5.3 .

where ps m y n. We claim that the quantity inside brackets in formula Ž5.3. is positive. This can be shown by grouping successive terms in pairs. Thus if p is even, the quantity is equal to

ž

1 2nq1 q

ž

y

1 2nq3

/ ž q

1 2 n q 2 py 3

y

1 2nq5

y

1 2nq7

1 2 n q 2 py 1

/

/

q ⭈⭈⭈

,

which is positive, since the difference inside each parenthesis is positive. If ps 1, the quantity is obviously positive, since it is then equal to 1rŽ2 n q 1.. If pG 3 is an odd integer, the quantity can be written as

ž

1 2nq1 q

ž

y

1 2nq3

/ ž q

1 2 n q 2 py 5

y

1 2nq5

y

1 2nq7

1 2 n q 2 py 3

/

q

/

q ⭈⭈⭈ 1

2 n q 2 py 1

,

139

INFINITE SEQUENCES

which is also positive. Hence, for any p, < am y an < s

1

y

2nq1

1 2nq3

q ⭈⭈⭈ q

Ž y1.

py1

2 n q 2 py 1

Ž 5.4 .

.

We now claim that 1

< am y an < -

2nq1

.

To prove this claim, let us again consider two cases. If p is even, then < am y an < s

1 2nq1 y

-

ž

y

ž

1 2nq3

1 2 n q 2 py 5

1 2nq1

1

y

2nq5

/

y ⭈⭈⭈

1

y

2 n q 2 py 3

/

y

1 2 n q 2 py 1

Ž 5.5 .

,

since all the quantities inside parentheses in Ž5.5. are positive. If p is odd, then < am y an < s

1 2nq1 y

ž

y

ž

1 2nq3

1 2 n q 2 py 3

y

y

1 2nq5

/

y ⭈⭈⭈

1 2 n q 2 py 1

/

-

1 2nq1

,

which proves our claim. On the basis of this result we can assert that for a given ⑀ ) 0, < am y an < - ⑀

if m ) n ) N,

where N is such that 1 2 Nq1

-⑀ ,

or equivalently, N)

1 2⑀

y

1 2

.

This shows that
a n4⬁ns1 is a Cauchy sequence.

140 EXAMPLE 5.1.7.

INFINITE SEQUENCES AND SERIES

Consider the sequence
a n4⬁ns1 , where a n s Ž y1 .

n

ž

1q

1 n

/

.

We have seen earlier that lim inf n™⬁ a n s y1 and lim sup n™⬁ a n s 1. Thus by Theorem 5.1.5 this sequence is not convergent. We can arrive at the same conclusion using the Cauchy criterion by showing that the ⑀-condition is not satisfied. This occurs whenever we can find an ⑀ ) 0 such that for however N may be chosen, < am y an < G ⑀ for some m ) N, n ) N. In our example, if N is any positive integer, then the inequality < am y an < G 2

Ž 5.6 .

can be satisfied by choosing m s ␯ and n s ␯ q 1, where ␯ is an odd integer greater than N.

5.2. INFINITE SERIES Let
a n4⬁ns1 be a given sequence. Consider the symbolic expression ⬁

Ý an s a1 q a2 q ⭈⭈⭈ qan q ⭈⭈⭈ .

Ž 5.7 .

ns1

By definition, this expression is called an infinite series, or just a series for simplicity, and a n is referred to as the nth term of the series. The finite sum n

sn s

Ý ai ,

n s 1, 2, . . . ,

is1

is called the nth partial sum of the series. Definition 5.2.1. sum Ž n s 1, 2, . . . ..

Consider the series Ý⬁ns1 a n . Let sn be its nth partial

1. The series is said to be convergent if the sequence
sn4⬁ns1 converges. In this case, if lim n™⬁ sn s s, where s is finite, then we say that the series converges to s, or that s is the sum of the series. Symbolically, this is

141

INFINITE SERIES

expressed by writing ss

⬁

Ý an .

ns1

2. If sn does not tend to a finite limit, then the series is said to be divergent. I Definition 5.2.1 formulates convergence of a series in terms of convergence of the associated sequence of its partial sums. By applying the Cauchy criterion ŽTheorem 5.1.6. to the latter sequence, we arrive at the following condition of convergence for a series: Theorem 5.2.1. The series Ý⬁ns1 a n , converges if and only if for a given ⑀ ) 0 there is an integer N such that n

Ý

ai - ⑀

for all n ) m ) N.

Ž 5.8 .

ismq1

Inequality Ž5.8. follows from applying Theorem 5.1.6 to the sequence
sn4⬁ns1 of partial sums of the series and noting that < sn y sm < s

n

Ý

ai

for n ) m.

ismq1

In particular, if n s m q 1, then inequality Ž5.8. becomes < a mq1 < - ⑀

Ž 5.9 .

for all m ) N. This implies that lim m ™⬁ a mq1 s 0, and hence lim n™⬁ a n s 0. We therefore conclude the following result: RESULT 5.2.1.

If Ý⬁ns1 a n is a convergent series, then lim n™⬁ a n s 0.

It is important here to note that the convergence of the nth term of a series to zero as n ™ ⬁ is a necessary condition for the convergence of the series. It is not, however, a sufficient condition, that is, if lim n™⬁ a n s 0, then it does not follow that Ý⬁ns1 a n converges. For example, as we shall see later, the series Ý⬁ns1 Ž1rn. is divergent, and its nth term goes to zero as n ™ ⬁. It is true, however, that if lim n™⬁ a n / 0, then Ý⬁ns1 a n is divergent. This follows from applying the law of contraposition to the necessary condition of convergence. We conclude the following: 1. If a n ™ 0 as n ™ ⬁, then no conclusion can be reached regarding convergence or divergence of Ý⬁ns1 a n .

142

INFINITE SEQUENCES AND SERIES

2. If a n ¢ 0 as n ™ ⬁, then Ý⬁ns1 a n is divergent. For example, the series Ý⬁ns1 w nrŽ n q 1.x is divergent, since n

lim

nq1

n™⬁

s 1 / 0.

EXAMPLE 5.2.1. One of the simplest series is the geometric series, Ý⬁ns1 a n. This series is divergent if < a < G 1, since lim n™⬁ a n / 0. It is convergent if < a < - 1 by the Cauchy criterion: Let n ) m. Then sn y sm s a mq1 q a mq2 q ⭈⭈⭈ qa n .

Ž 5.10 .

By multiplying the two sides of Ž5.10. by a, we get a Ž sn y sm . s a mq2 q a mq3 q ⭈⭈⭈ qa nq1 .

Ž 5.11 .

If we now subtract Ž5.11. from Ž5.10., we obtain a mq1 y a nq1

sn y sm s

1ya

.

Ž 5.12 .

Since < a < - 1, we can find an integer N such that for m ) N, n ) N, < a < mq1 < a < nq1 -

⑀ Ž 1 y a. 2

⑀ Ž 1 y a. 2

, .

Hence, for a given ⑀ ) 0, < sn y sm < - ⑀

if n ) m ) N.

Formula Ž5.12. can actually be used to find the sum of the geometric series when < a < - 1. Let m s 1. By taking the limits of both sides of Ž5.12. as n ™ ⬁ we get lim sn s s1 q

n™⬁

s aq s

a2 1ya a2 1ya

a 1ya

.

,

since lim a nq1 s 0, n™⬁

143

INFINITE SERIES

EXAMPLE 5.2.2. Consider the series Ý⬁ns1 Ž1rn!.. This series converges by the Cauchy criterion. To show this, we first note that n!s n Ž n y 1 . Ž n y 2 . = ⭈⭈⭈ = 3 = 2 = 1 G 2 ny1

for n s 1, 2, . . . .

Hence, for n ) m, < sn y sm < s

1

Ž m q 1. ! 1

F

2

m

1

q 2 n

s2

Ý

q

mq1

1

ismq1

2i

1

Ž m q 2. !

q ⭈⭈⭈ q

q ⭈⭈⭈ q

1 n!

1 2

ny1

.

This is a partial sum of a convergent geometric series with as 12 - 1 wsee formula Ž5.10.x. Consequently, < sn y sm < can be made smaller than any given ⑀ ) 0 by choosing m and n large enough. Theorem 5.2.2. If Ý⬁ns1 a n and Ý⬁ns1 bn are two convergent series, and if c is a constant, then the following series are also convergent: 1. Ý⬁ns1 Ž ca n . s cÝ⬁ns1 a n . 2. Ý⬁ns1 Ž a n q bn . s Ý⬁ns1 a n q Ý⬁ns1 bn . I

Proof. The proof is left to the reader. Definition 5.2.2. is convergent.

The series Ý⬁ns1 a n is absolutely convergent if Ý⬁ns1 < a n < I

For example, the series Ý⬁ns1 wŽy1. nrn!x is absolutely convergent, since is convergent, as was seen in Example 5.2.2.

Ý⬁ns1 Ž1rn!.

Theorem 5.2.3.

Every absolutely convergent series is convergent.

Proof. Consider the series Ý⬁ns1 a n , and suppose that Ý⬁ns1 < a n < is convergent. We have that n

Ý

ismq1

n

ai F

Ý

ismq1

< ai < .

Ž 5.13 .

144

INFINITE SEQUENCES AND SERIES

By applying the Cauchy criterion to Ý⬁ns1 < a n < we can find an integer N such that for a given ⑀ ) 0, n

Ý

< ai < - ⑀

if n ) m ) N.

Ž 5.14 .

ismq1

From Ž5.13. and Ž5.14. we conclude that Ý⬁ns1 a n satisfies the Cauchy criterion and is therefore convergent by Theorem 5.2.1. I Note that it is possible that Ý⬁ns1 a n is convergent while Ý⬁ns1 < a n < is divergent. In this case, the series Ý⬁ns1 a n is said to be conditionally convergent. Examples of this kind of series will be seen later. In the next section we shall discuss convergence of series whose terms are positive. 5.2.1. Tests of Convergence for Series of Positive Terms Suppose that the terms of the series Ý⬁ns1 a n are such that a n ) 0 for n ) K, where K is a constant. Without loss of generality we shall consider that K s 1. Such a series is called a series of positive terms. Series of positive terms are interesting because the study of their convergence is comparatively simple and can be used in the determination of convergence of more general series whose terms are not necessarily positive. It is easy to see that a series of positive terms diverges if and only if its sum is q⬁. In what follows we shall introduce techniques that simplify the process of determining whether or not a given series of positive terms is convergent. We refer to these techniques as tests of convergence. The advantage of these tests is that they are in general easier to apply than the Cauchy criterion. This is because evaluating or obtaining inequalities involving the expression Ý nismq1 a i in Theorem 5.2.1 can be somewhat difficult. The tests of convergence, however, have the disadvantage that they can sometime fail to determine convergence or divergence, as we shall soon find out. It should be remembered that these tests apply only to series of positive terms. The Comparison Test This test is based on the following theorem: Theorem 5.2.4. Let Ý⬁ns1 a n and Ý⬁ns1 bn be two series of positive terms such that a n F bn for n ) N0 , where N0 is a fixed integer. i. If Ý⬁ns1 bn converges, then so does Ý⬁ns1 a n . ii. If Ý⬁ns1 a n is divergent, then Ý⬁ns1 bn is divergent too.

145

INFINITE SERIES

Proof. We have that n

Ý

n

Ý

ai F

ismq1

Ž 5.15 .

for n ) m ) N0 .

bi

ismq1

If Ý⬁ns1 bn is convergent, then for a given ⑀ ) 0 there exists an integer N1 such that n

Ý

bi - ⑀

Ž 5.16 .

for n ) m ) N1 .

ismq1

From Ž5.15. and Ž5.16. it follows that if n ) m ) N, where N s maxŽ N0 , N1 ., then n

Ý

ai - ⑀ ,

ismq1

which proves Ži.. The proof of Žii. follows from applying the law of contraposition to Ži.. I To determine convergence or divergence of Ý⬁ns1 a n we thus need to have in our repertoire a collection of series of positive terms whose behavior Žwith regard to convergence or divergence. is known. These series can then be compared against Ý⬁ns1 a n . For this purpose, the following series can be useful: a. Ý⬁ns1 1rn. This is a divergent series called the harmonic series. b. Ý⬁ns1 1rn k . This is divergent if k - 1 and is convergent if k ) 1. To prove that the harmonic series is divergent, let us consider its nth partial sum, namely, n

sn s

Ý

is1

1

.

i

Let A ) 0 be an arbitrary positive number. Choose n large enough so that n ) 2 m , where m ) 2 A. Then for such values of n,

ž

sn ) 1 q q

)

1 2

ž

q

1 2

/ ž q

1 3

1 2

my1

2 4

q

q1 4 8

q

1 4

/ ž q

q ⭈⭈⭈ q

q ⭈⭈⭈ q

1 2m

2 my1 2

m

1

q

5

1 6

q

1 7

/ s

m 2

) A.

q

1 8

/

q ⭈⭈⭈

Ž 5.17 .

146

INFINITE SEQUENCES AND SERIES

Since A is arbitrary and sn is a monotone increasing function of n, inequality Ž5.17. implies that sn ™ ⬁ as n ™ ⬁. This proves divergence of the harmonic series. Let us next consider the series in Žb.. If k - 1, then 1rn k ) 1rn and ⬁ Ý ns1 Ž1rn k . must be divergent by Theorem 5.2.4Žii.. Suppose now that k ) 1. Consider the nth partial sum of the series, namely, sXn s

n

1

Ý

is1

.

ik

Then, by choosing m large enough so that 2 m ) n we get sXn F

2 my1

1

Ý

ik

is1

s1q q

F1q q

s1q

ž

1 2

k

1

q

3

k

1

ž

1 2

k

1

q

2

1

Ž2

my1

k

.

1 4

q ⭈⭈⭈ q

k

Ž 2 my1 .

/ ž q

k

/ ž q

k

q ⭈⭈⭈ q

5

q

k

6

Ž 2 m y 1. k

q

1 4

1 4

q

1 7k

/

q ⭈⭈⭈

k

q

1 4k

/

q ⭈⭈⭈

1

Ž2

my1

.

k

k

4

2 my1

2

4

Ž 2 my1 .

q ⭈⭈⭈ q k

k

k

q

k

2

q k

1

1

1 4

1

q

m

s

Ý aiy1 ,

Ž 5.18 .

is1

where as 1r2 ky1 . But the right-hand side of Ž5.18. represents the mth partial sum of a convergent geometric series Žsince a- 1.. Hence, as m ™ ⬁, the right-hand side of Ž5.18. converges to ⬁

1

Ý aiy1 s 1 y a

Ž see Example 5.2.1 . .

is1

Thus the sequence
sXn4⬁ns1 is bounded. Since it is also monotone increasing, it must be convergent Žsee Theorem 5.1.2.. This proves convergence of the series Ý⬁ns1 Ž1rn k . for k ) 1.

147

INFINITE SERIES

Another version of the comparison test in Theorem 5.2.4 that is easier to implement is given by the following theorem: Theorem 5.2.5. Let Ý⬁ns1 a n and Ý⬁ns1 bn be two series of positive terms. If there exists a positive constant l such that a n and lbn are asymptotically equal, a n ; lbn as n ™ ⬁ Žsee Section 3.3., that is, an

lim

bn

n™⬁

s l,

then the two series are either both convergent or both divergent. Proof. There exists an integer N such that an

yl -

bn

l

if n ) N,

2

or equivalently, l 2

-

an bn

-

3l

whenever n ) N.

2

If Ý⬁ns1 a n is convergent, then Ý⬁ns1 bn is convergent by a combination of Theorem 5.2.2Ž1. and 5.2.4Ži., since bn - Ž2rl . a n . Similarly, if Ý⬁ns1 bn converges, then so does Ý⬁ns1 a n , since a n - Ž3lr2. bn . If Ý⬁ns1 a n is divergent, then Ý⬁ns1 bn is divergent too by a combination of Theorems 5.2.2Ž1. and 5.2.4Žii., since bn ) Ž2r3l . a n . Finally, Ý⬁ns1 a n diverges if the same is true of Ý⬁ns1 bn , since a n ) Ž lr2. bn . I EXAMPLE 5.2.3. The series Ý⬁ns1 Ž n q 2.rŽ n3 q 2 n q 1. is convergent, since nq2 n q2nq1 3

;

1 n2

as n ™ ⬁,

which is the nth term of a convergent series wrecall that Ý⬁ns1 Ž1rn k . is convergent if k ) 1x.

'

EXAMPLE 5.2.4. Ý⬁ns1 1r n Ž n q 1 . is divergent, because 1

'n Ž n q 1.

;

1 n

as n ™ ⬁,

which is the nth term of the divergent harmonic series.

148

INFINITE SEQUENCES AND SERIES

The Ratio or d’Alembert’s Test This test is usually attributed to the French mathematician Jean Baptiste d’Alembert Ž1717᎐1783., but is also known as Cauchy’s ratio test after Augustin-Louis Cauchy Ž1789᎐1857.. Theorem 5.2.6. following hold:

Let Ý⬁ns1 a n be a series of positive terms. Then the

1. The series converges if lim sup n™⬁Ž a nq1 ra n . - 1 Žsee Definition 5.1.2.. 2. The series diverges if lim inf n™⬁Ž a nq1 ra n . ) 1 Žsee Definition 5.1.2.. 3. If lim inf n™⬁Ž a nq1 ra n . F 1 F lim sup n™⬁Ž a nq1 ra n ., no conclusion can be made regarding convergence or divergence of the series Žthat is, the ratio test fails.. In particular, if lim n™⬁Ž a nq1 ra n . s r exists, then the following hold: 1. The series converges if r - 1. 2. The series diverges if r ) 1. 3. The test fails if r s 1. Proof. Let ps lim inf n™⬁Ž a nq1 ra n ., q s lim sup n™⬁Ž a nq1 ra n .. 1. If q - 1, then by the definition of the upper limit ŽDefinition 5.1.2., there exists an integer N such that a nq1 an

- q⬘

for n G N,

Ž 5.19 .

where q⬘ is chosen such that q - q⬘ - 1. ŽIf a nq1 ra n G q⬘ for infinitely many values of n, then the sequence
a nq1 ra n4⬁ns1 has a subsequential limit greater than or equal to q⬘, which exceeds q. This contradicts the definition of q.. From Ž5.19. we then get a Nq1 - a N q⬘ a Nq2 - a Nq1 q⬘ - a N q⬘ 2 , . . . a Nqm - a Nqmy1 q⬘ - a N q⬘ m , where m G 1. Thus for n ) N, a n - a N q⬘Ž nyN . s a N Ž q⬘ .

yN

q⬘ n .

149

INFINITE SERIES

Hence, the series converges by comparison with the convergent geometric series Ý⬁ns1 q⬘ n, since q⬘ - 1. 2. If p) 1, then in an analogous manner we can find an integer N such that a nq1 an

) p⬘

for n G N,

Ž 5.20 .

where p⬘ is chosen such that p) p⬘ ) 1. But this implies that a n cannot tend to zero as n ™ ⬁, and the series is therefore divergent by Result 5.2.1. 3. If pF 1 F q, then we can demonstrate by using an example that the ratio test is inconclusive: Consider the two series Ý⬁ns1 Ž1rn., Ý⬁ns1 Ž1rn2 .. For both series, ps q s 1 and hence pF 1 F q, since lim n™⬁Ž a nq1 ra n . s 1. But the first series is divergent while the second is convergent, as was seen earlier. I EXAMPLE 5.2.5. Consider the same series as in Example 5.2.2. This series was shown to be convergent by the Cauchy criterion. Let us now apply the ratio test. In this case,

lim

n™⬁

a nq1 an

s lim

n™⬁

s lim

n™⬁

1

1

Ž n q 1. !

n!

1 nq1

s 0 - 1,

which indicates convergence by Theorem 5.2.6Ž1.. Nurcombe Ž1979. stated and proved the following extension of the ratio test: Theorem 5.2.7. positive integer.

Let Ý⬁ns1 a n be a series of positive terms, and k be a fixed

1. If lim n™⬁Ž a nqk ra n . - 1, then the series converges. 2. If lim n™⬁Ž a nqk ra n . ) 1, then the series diverges. This test reduces to the ratio test when k s 1. The Root or Cauchy’s Test This is a more powerful test than the ratio test. It is based on the following theorem:

150

INFINITE SEQUENCES AND SERIES

Theorem 5.2.8. Let Ý⬁ns1 a n be a series of positive terms. Let n lim sup n™⬁ a1r s ␳ . Then we have the following: n 1. The series converges if ␳ - 1. 2. The series diverges if ␳ ) 1. 3. The test is inconclusive if ␳ s 1. n In particular, if lim n™⬁ a1r s ␶ exists, then we have the following: n

1. The series converges if ␶ - 1. 2. The series diverges if ␶ ) 1. 3. The test is inconclusive if ␶ s 1. Proof. 1. As in Theorem 5.2.6Ž1., if ␳ - 1, then there is an integer N such that n a1r -␳⬘ n

for n G N,

where ␳ ⬘ is chosen such that ␳ - ␳ ⬘ - 1. Thus an - ␳ ⬘n

for n G N.

The series is therefore convergent by comparison with the convergent geometric series Ý⬁ns1 ␳ ⬘ n, since ␳ ⬘ - 1. 2. Suppose that ␳ ) 1. Let ⑀ ) 0 be such that ⑀ - ␳ y 1. Then n a1r )␳y⑀)1 n

for infinitely many values of n Žwhy?.. Thus for such values of n, n

an ) Ž ␳ y ⑀ . , which implies that a n cannot tend to zero as n ™ ⬁ and the series is therefore divergent by Result 5.2.1. 3. Consider again the two series Ý⬁ns1 Ž1rn., Ý⬁ns1 Ž1rn2 .. In both cases ␳ s 1 Žsee Exercise 5.18.. The test therefore fails, since the first series is divergent and the second is convergent. I NOTE 5.2.1. We have mentioned earlier that the root test is more powerful than the ratio test. By this we mean that whenever the ratio test shows convergence or divergence, then so does the root test; whenever the root test is inconclusive, the ratio test is inconclusive too. However, there are situations where the ratio test fails, but the root test doe not Žsee Example 5.2.6.. This fact is based on the following theorem:

151

INFINITE SERIES

Theorem 5.2.9. lim inf n™⬁

If a n ) 0, then

a nq1 an

n n F lim inf a1r F lim sup a1r F lim sup n n

n™⬁

n™⬁

n™⬁

a nq1 an

.

Proof. It is sufficient to prove the two inequalities n lim sup a1r F lim sup n

n™⬁

a nq1

n™⬁

a nq1

lim inf

an

n™⬁

an

,

n F lim inf a1r n .

n™⬁

Ž 5.21 . Ž 5.22 .

Inequality Ž5.21.: Let q s lim sup n™⬁Ž a nq1 ra n .. If q s ⬁, then there is nothing to prove. Let us therefore consider that q is finite. If we choose q⬘ such that q - q⬘, then as in the proof of Theorem 5.2.6Ž1., we can find an integer N such that a n - a N Ž q⬘ .

yN

q⬘ n

for n ) N.

Hence, n a1r - a N Ž q⬘ . n

yN

1rn

q⬘.

Ž 5.23 .

As n ™ ⬁, the limit of the right-hand side of inequality Ž5.23. is q⬘. It follows that n F q⬘. lim sup a1r n

n™⬁

Ž 5.24 .

Since Ž5.24. is true for any q⬘ ) q, then we must also have n F q. lim sup a1r n

n™⬁

Inequality Ž5.22.: Let ps lim inf n™⬁Ž a nq1 ra n .. We can consider p to be finite Žif ps ⬁, then q s ⬁ and the proof of the theorem will be complete; if ps y⬁, then there is nothing to prove.. Let p⬘ be chosen such that p⬘ - p. As in the proof of Theorem 5.2.6Ž2., we can find an integer N such that a nq1 an

) p⬘

for n G N.

From Ž5.25. it is easy to show that a n ) a N Ž p⬘ .

yN

p⬘ n

for n G N.

Ž 5.25 .

152

INFINITE SEQUENCES AND SERIES

Hence, for such values of n, n a1r ) a N Ž p⬘ . n

yN

1rn

p⬘.

Consequently,

Ž 5.26 .

n lim inf a1r G p⬘. n

n™⬁

Since Ž5.26. is true for any p⬘ - p, then n G p. lim inf a1r n

n™⬁

From Theorem 5.2.9 we can easily see that whenever q - 1, then n n lim sup n™⬁ a1r - 1; whenever p) 1, then lim sup n™⬁ a1r ) 1. In both cases, n n if convergence or divergence of the series is resolved by the ratio test, then it can also be resolved by the root test. If, however, the root test fails Žwhen n lim sup n™⬁ a1r s 1., then the ratio test fails too by Theorem 5.2.6Ž3.. On the n other hand, it is possible for the ratio test to be inconclusive whereas the root test is not. This occurs when

lim inf n™⬁

a nq1 an

n n F lim inf a1r F lim sup a1r - 1 F lim sup n n

n™⬁

n™⬁

n™⬁

a nq1 an

.

I

EXAMPLE 5.2.6. Consider the series Ý⬁ns1 Ž a n q b n ., where 0 - a- b- 1. This can be written as Ý⬁ns1 c n , where for n G 1,

cn s

½

aŽ nq1. r2 b n r2

if n is odd, if n is even.

Now, c nq1 cn

s

n c 1r s n

½

½

Ž

.

Ž bra. nq1 r2 nr2 a Ž arb . aŽ nq1. rŽ2 n. b1r2

if n is odd, if n is even,

if n is odd, if n is even.

153

INFINITE SERIES

n As n ™ ⬁, c nq1 rc n has two limits, namely 0 and ⬁; c 1r has two limits, a1r2 n 1r2 and b . Thus

lim inf n™⬁

lim sup n™⬁

c nq1 cn c nq1 cn

s 0, s ⬁,

n lim sup c 1r s b1r2 - 1. n

n™⬁

Since 0 F 1 F ⬁, we can clearly see that the ratio test is inconclusive, whereas the root test indicates that the series is convergent. Maclaurin’s (or Cauchy’s) Integeral Test This test was introduced by Colin Maclaurin Ž1698᎐1746. and then rediscovered by Cauchy. The description and proof of this test will be given in Chapter 6. Cauchy’s Condensation Test Let us consider the following theorem: Theorem 5.2.10. Let Ý⬁ns1 a n be a series of positive terms, where a n is a monotone decreasing function of n Žs 1, 2, . . . .. Then Ý⬁ns1 a n converges or diverges if and only if the same is true of the series Ý⬁ns1 2 na2 n . Proof. Let sn and t m be the nth and mth partial sums, respectively, of Ý⬁ns1 a n and Ý⬁ns1 2 na2 n . If m is such that n - 2 m , then sn F a1 q Ž a2 q a3 . q Ž a4 q a5 q a6 q a7 . q ⭈⭈⭈ q Ž a2 m q a2 m q1 q ⭈⭈⭈ qa2 m q2 m y1 . F a1 q 2 a2 q 4 a4 q ⭈⭈⭈ q2 m a2 m s t m .

Ž 5.27 .

Furthermore, if n ) 2 m , then sn G a1 q a2 q Ž a3 q a4 . q ⭈⭈⭈ q Ž a2 my1 q1 q ⭈⭈⭈ qa2 m . G

a1 2

q a2 q 2 a4 q ⭈⭈⭈ q2 my1 a2 m s

tm 2

.

Ž 5.28 .

If Ý⬁ns1 2 na2 n diverges, then t m ™ ⬁ as m ™ ⬁. Hence, from Ž5.28., sn ™ ⬁ as n ™ ⬁, and the series Ý⬁ns1 a n is also divergent.

154

INFINITE SEQUENCES AND SERIES

Now, if Ý⬁ns1 2 na2 n converges, then the sequence
t m 4⬁ms1 is bounded. From Ž5.27., the sequence
sn4⬁ns1 is also bounded. It follows that Ý⬁ns1 a n is a convergent series Žsee Exercise 5.13.. I EXAMPLE 5.2.7. Consider again the series Ý⬁ns1 Ž1rn k .. We have already seen that this series converges if k ) 1 and diverges if k F 1. Let us now apply Cauchy’s condensation test. In this case, ⬁

Ý 2 na2

n

⬁

ns1

⬁

1

1

Ý 2 n 2 nk s Ý

s

ns1

2

ns1

nŽ ky1.

is a geometric series Ý⬁ns1 b n, where bs 1r2 ky1. If k F 1, then bG 1 and the series diverges. If k ) 1, then b- 1 and the series converges. It is interesting to note that in this example, both the ratio and the root tests fail. The following tests enable us to handle situations where the ratio test fails. These tests are particular cases on a general test called Kummer’s test. Kummer’s Test This test is named after the German mathematician Ernst Eduard Kummer Ž1810᎐1893.. Theorem 5.2.11. Let Ý⬁ns1 a n and Ý⬁ns1 bn be two series of positive terms. Suppose that the series Ý⬁ns1 bn is divergent. Let

lim

n™⬁

ž

an

1

bn a nq1

y

1 bnq1

/

s ␭,

Then Ý⬁ns1 a n converges if ␭ ) 0 and diverges if ␭ - 0. Proof. Suppose that ␭ ) 0. We can find an integer N such that for n ) N, 1

an

bn a nq1

y

1 bnq1

)

␭ 2

Ž 5.29 .

.

Inequality Ž5.29. can also be written as

a nq1 -

2

␭

ž

an bn

y

a nq1 bnq1

/

.

Ž 5.30 .

155

INFINITE SERIES

If sn is the nth partial sum of Ý⬁ns1 a n , then from Ž5.30. and for n ) N, snq1 - sNq1 q

nq1

2

␭

Ý

isNq2

ž

a iy1 biy1

y

ai bi

/

,

that is, snq1 - sNq1 q snq1 - sNq1 q

ž

2

a Nq1

␭ bNq1

y

2 a Nq1

a nq1 bnq1

/

,

Ž 5.31 .

for n ) N.

␭ bNq1

Inequality Ž5.31. indicates that the sequence
sn4⬁ns1 is bounded. Hence, the series Ý⬁ns1 a n is convergent Žsee Exercise 5.13.. Now, let us suppose that ␭ - 0. We can find an integer N such that 1

an

bn a nq1

y

1 bnq1

-0

for n ) N.

Thus for such values of n, a nq1 )

an bn

Ž 5.32 .

bnq1 .

It is easy to verify that because of Ž5.32., an )

a Nq1 bNq1

Ž 5.33 .

bn

for n G N q 2. Since Ý⬁ns1 bn is divergent, then from Ž5.33. and the use of the comparison test we conclude that Ý⬁ns1 a n is divergent too. I Two particular cases of Kummer’s test are Raabe’s test and Gauss’s test. Raabe’s Test This test was established in 1832 by J. L. Raabe. Theorem 5.2.12. that

Suppose that Ý⬁ns1 a n is a series of positive terms and an a nq1

s1q

␶ n

qo

ž / 1

n

as n ™ ⬁.

Then Ý⬁ns1 a n converges if ␶ ) 1 and diverges if ␶ - 1.

156

INFINITE SEQUENCES AND SERIES

Proof. We have that an a nq1

s1q

␶ n

qo

ž /

/

™0

1

n

.

This means that n

ž

an a nq1

y1y

␶ n

Ž 5.34 .

as n ™ ⬁. Equivalently, Ž5.34. can be expressed as lim

n™⬁

ž

nan a nq1

/

y n y 1 s ␶ y 1.

Ž 5.35 .

Let bn s 1rn in Ž5.35.. This is the nth term of a divergent series. If we now apply Kummer’s test, we conclude that the series Ý⬁ns1 a n converges if ␶ y 1 ) 0 and diverges if ␶ y 1 - 0. I Gauss’s Test This test is named after Carl Friedrich Gauss Ž1777᎐1855.. It provides a slight improvement over Raabe’s test in that it usually enables us to handle the case ␶ s 1. For such a value of ␶ , Raabe’s test is inconclusive. Theorem 5.2.13.

Let Ý⬁ns1 a n be a series of positive terms. Suppose that an a nq1

s1q

␪ n

qO

ž / 1

n

␦q1

␦ ) 0.

,

Then Ý⬁ns1 a n converges if ␪ ) 1 and diverges if ␪ F 1. Proof. Since O

ž / ž / 1

n

so

␦q1

1

n

,

then by Raabe’s test, Ý⬁ns1 a n converges if ␪ ) 1 and diverges if ␪ - 1. Let us therefore consider ␪ s 1. We have an a nq1

s1q

1 n

qO

ž / 1

n

␦q1

.

157

INFINITE SERIES

Put bn s 1rŽ n log n., and consider lim

n™⬁

ž

an

1

bn a nq1

s lim

n™⬁

½

y

1 bnq1

/

n log n 1 q

s lim Ž n q 1 . log n™⬁

1 n

qO

n nq1

ž / 1

n

␦q1

y Ž n q 1 . log Ž n q 1 .

q Ž n log n . O

ž / 1

n

␦q1

5

s y1.

This is true because lim Ž n q 1 . log

n™⬁

n nq1

Ž by l’Hospital’s rule .

s y1

and lim Ž n log n . O

n™⬁

ž / 1

n

␦q1

s0

see Example 4.2.3 Ž 2 . .

Since Ý⬁ns1 w1rŽ n log n.x is a divergent series Žthis can be shown by using Cauchy’s condensation test., then by Kummer’s test, the series Ý⬁ns1 a n is divergent. I EXAMPLE 5.2.8. Gauss established his test in order to determine the convergence of the so-called hypergeometric series. He managed to do so in an article published in 1812. This series is of the form 1 q Ý⬁ns1 a n , where an s

␣ Ž ␣ q 1 . Ž ␣ q 2 . ⭈⭈⭈ Ž ␣ q n y 1 . ␤ Ž ␤ q 1 . Ž ␤ q 2 . ⭈⭈⭈ Ž ␤ q n y 1 . n!␥ Ž ␥ q 1 . Ž ␥ q 2 . ⭈⭈⭈ Ž ␥ q n y 1 .

,

n s 1, 2, . . . , where ␣ , ␤ , ␥ are real numbers, and none of them is zero or a negative integer. We have an a nq1

s

n2 q Ž ␥ q 1 . n q ␥ Ž n q 1. Ž n q ␥ . s 2 Ž n q ␣ . Ž n q ␤ . n q Ž ␣ q ␤ . n q ␣␤

s1q

␥q1y␣y␤ n

qO

ž / 1

n2

.

In this case, ␪ s ␥ q 1 y ␣ y ␤ and ␦ s 1. By Gauss’s test, this series is convergent if ␪ ) 1, or ␥ ) ␣ q ␤ , and is divergent if ␪ F 1, or ␥ F ␣ q ␤ .

158

INFINITE SEQUENCES AND SERIES

5.2.2. Series of Positive and Negative Terms Consider the series Ý⬁ns1 a n , where a n may be positive or negative for n G 1. The convergence of this general series can be determined by the Cauchy criterion ŽTheorem 5.1.6.. However, it is more convenient to consider the series Ý⬁ns1 < a n < of absolute values, to which the tests of convergence in Section 5.2.1 can be applied. We recall from Definition 5.2.2 that if the latter series converges, then the series Ý⬁ns1 a n is absolutely convergent. This is a stronger type of convergence than the one given in Definition 5.2.1, since by Theorem 5.2.3 convergence of Ý⬁ns1 < a n < implies convergence of Ý⬁ns1 a n . The converse, however, is not necessarily true, that is, convergence of Ý⬁ns1 a n does not necessarily imply convergence of Ý⬁ns1 < a n < . For example, consider the series

Ž y1.

⬁

Ý

ny1

2ny1

ns1

s1y

1 3

q

1 5

y

1 7

Ž 5.36 .

q ⭈⭈⭈ .

This series is convergent by the result of Example 5.1.6. It is not, however, absolutely convergent, since Ý⬁ns1 w1rŽ2 n y 1.x is divergent by comparison with the harmonic series Ý⬁ns1 Ž1rn., which is divergent. We recall that a series such as Ž5.36. that converges, but not absolutely, is called a conditionally convergent series. The series in Ž5.36. belongs to a special class of series known as alternating series. Definition 5.2.3. The series Ý⬁ns1 Žy1. ny1 a n , where a n ) 0 for n G 1, is called an alternating series. I The following theorem, which was established by Gottfried Wilhelm Leibniz Ž1646᎐1716., can be used to determine convergence of alternating series: Theorem 5.2.14. Let Ý⬁ns1 Žy1. ny1 a n be an alternating series such that the sequence
a n4⬁ns1 is monotone decreasing and converges to zero as n ™ ⬁. Then the series is convergent. Proof. Let sn be the nth partial sum of the series, and let m be an integer such that m - n. Then n

sn y sm s

Ý Ž y1. iy1 ai

ismq1

s Ž y1 .

m

a mq1 y a mq2 q ⭈⭈⭈ q Ž y1 .

nymy1

an .

Ž 5.37 .

159

INFINITE SERIES

Since
a n4⬁ns1 is monotone decreasing, it is easy to show that the quantity inside brackets in Ž5.37. is nonnegative. Hence, < sn y sm < s a mq1 y a mq2 q ⭈⭈⭈ q Ž y1 . nymy1 a n . Now, if n y m is odd, then < sn y sm < s a mq1 y Ž a mq2 y a mq3 . y ⭈⭈⭈ y Ž a ny1 y a n . F a mq1 . If n y m is even, then < sn y sm < s a mq1 y Ž a mq2 y a mq3 . y ⭈⭈⭈ y Ž a ny2 y a ny1 . y a n F a mq1 . Thus in both cases < sn y sm < F a mq1 . Since the sequence
a n4⬁ns1 converges to zero, then for a given ⑀ ) 0 there exists an integer N such that for m G N, a mq1 - ⑀ . Consequently, < sn y sm < - ⑀

if n ) m G N.

By Theorem 5.2.1, the alternating series is convergent.

I

EXAMPLE 5.2.9. The series given by formula Ž5.36. was shown earlier to be convergent. This result can now be easily verified with the help of Theorem 5.2.14. EXAMPLE 5.2.10. The series Ý⬁ns1 Žy1. nrn k is absolutely convergent if k ) 1, is conditionally convergent if 0 - k F 1, and is divergent if k F 0 Žsince the nth term does not go to zero.. EXAMPLE 5.2.11. The series Ý⬁ns2 Žy1. nrŽ'n log n. is conditionally convergent, since it converges by Theorem 5.2.14, but the series of absolute values diverges by Cauchy’s condensation test ŽTheorem 5.2.10.. 5.2.3. Rearrangement of Series One of the main differences between infinite series and finite series is that whereas the latter are amenable to the laws of algebra, the former are not necessarily so. In particular, if the order of terms of an infinite series is altered, its sum Žassuming it converges. may, in general, change; or worse, the

160

INFINITE SEQUENCES AND SERIES

altered series may even diverge. Before discussing this rather disturbing phenomenon, let us consider the following definition: Definition 5.2.4. Let Jq denote the set of positive integers and Ý⬁ns1 a n be a given series. Then a second series such as Ý⬁ns1 bn is said to be a rearrangement of Ý⬁ns1 a n if there exists a one-to-one and onto function f : Jq™ Jq such that bn s a f Ž n. for n G 1. For example, the series

Ž 5.38 .

1 q 13 y 12 q 15 q 17 y 14 q ⭈⭈⭈ ,

where two positive terms are followed by one negative term, is a rearrangement of the alternating harmonic series

Ž 5.39 .

1 y 12 q 13 y 14 q 15 y ⭈⭈⭈ .

The series in Ž5.39. is conditionally convergent, as is the series in Ž5.38.. However, the two series have different sums Žsee Exercise 5.21.. I Fortunately, for absolutely convergent series we have the following theorem: Theorem 5.2.15. If the series Ý⬁ns1 a n is absolutely convergent, then any rearrangement of it remains absolutely convergent and has the same sum. Proof. Suppose that Ý⬁ns1 a n is absolutely convergent and that Ý⬁ns1 bn is a rearrangement of it. By Theorem 5.2.1, for a given ⑀ ) 0, there exists an integer N such that for all n ) m ) N, n

Ý

< ai < -

ismq1

⑀ 2

.

We then have ⬁

Ý

< a mqk < F

ks1

⑀

if m ) N.

2

Now, let us choose an integer M large enough so that


1, 2, . . . , N q 1 4 ;
f Ž 1 . , f Ž 2 . , . . . , f Ž M . 4 . It follows that if n ) M, then f Ž n. G N q 2. Consequently, for n ) m ) M, n

Ý

< bi < s

ismq1

n

Ý

< a f Ž i. <

ismq1

F

⬁

Ý

ks1

< a Nqkq1 < F

⑀ 2

.

161

INFINITE SERIES

This implies that the series Ý⬁ns1 < bn < satisfies the Cauchy criterion of Theorem 5.2.1. Therefore, Ý⬁ns1 bn is absolutely convergent. We now show that the two series have the same sum. Let s s Ý⬁ns1 a n , and sn be its nth partial sum. Then, for a given ⑀ ) 0 there exists an integer N large enough so that < sNq1 y s < -

⑀

.

2

If t n is the nth partial sum of Ý⬁ns1 bn , then < t n y s < F < t n y sNq1 < q < sNq1 y s < . By choosing M large enough as was done earlier, and by taking n ) M, we get n

Nq1

is1

is1

Ý bi y Ý

< t n y sNq1 < s

ai

n

Nq1

is1

is1

Ý a f Ž i. y Ý

s

⬁

Ý

F

ai

< a Nqkq1 < F

ks1

⑀ 2

,

since if n ) M,


a1 , a2 , . . . , aNq1 4 ;
a f Ž1. , a f Ž2. , . . . , a f Ž n. 4 . Hence, for n ) M, < tn y s < - ⑀ , which shows that the sum of the series Ý⬁ns1 bn is s.

I

Unlike absolutely convergent series, those that are conditionally convergent are susceptible to rearrangements of their terms. To demonstrate this, let us consider the following alternating series: ⬁

⬁

ns1

ns1

Ý an s Ý

Ž y1.

ny1

.

'n

This series is conditionally convergent, since it is convergent by Theorem 5.2.14 while Ý⬁ns1 Ž1r 'n . is divergent. Let us consider the following rearrangement: ⬁

1

Ý bn s 1 q '3

ns1

y

1

'2

q

1

'5

q

1

'7

y

1

'4

q ⭈⭈⭈

Ž 5.40 .

162

INFINITE SEQUENCES AND SERIES

in which two positive terms are followed by one that is negative. Let sX3 n denote the sum of the first 3n terms of Ž5.40.. Then

ž

sX3 n s 1 q q

ž

'3

ž'

y

1

'2

1

4ny3

s 1y q

1

1

'2

1

q

y

3

1

1

q

'2 n q 1

y

'4 n y 1

1

q

q

5

/ ž'

'2 n q 1

s s2 n q

/ ž'

1

q

1

'2 n q 3 q

/

1

'4

1

'7

y

1

'4

/ ž'

1

'2 n

q ⭈⭈⭈ q

q ⭈⭈⭈ q

'2 n q 3

1

/

q ⭈⭈⭈

1

y

2ny1

1

'4 n y 3

q ⭈⭈⭈ q

1

q

'4 n y 1

1

'2 n

/

1

'4 n y 1 ,

where s2 n is the sum of the first 2 n terms of the original series. We note that sX3 n ) s2 n q

n

'4 n y 1

.

Ž 5.41 .

If s is the sum of the original series, then lim n™⬁ s2 n s s in Ž5.41.. But, since lim

n™⬁

n

'4 n y 1

s ⬁,

the sequence
sX3 n4⬁ns1 is not convergent, which implies that the series in Ž5.40. is divergent. This clearly shows that a rearrangement of a conditionally convergent series can change its character. This rather unsettling characteristic of conditionally convergent series is depicted in the following theorem due to Georg Riemann Ž1826᎐1866.: Theorem 5.2.16. A conditionally convergent series can always be rearranged so as to converge to any given number s, or to diverge to q⬁ or to y⬁. Proof. The proof can be found in several books, for example, Apostol Ž1964, page 368., Fulks Ž1978, page 489., Knopp Ž1951, page 318., and Rudin Ž1964, page 67.. I 5.2.4. Multiplication of Series Suppose that Ý⬁ns1 a n and Ý⬁ns1 bn are two series. We recall from Theorem 5.2.2 that if these series are convergent, then their sum is a convergent series

163

INFINITE SERIES

obtained by adding the two series term by term. The product of these two series, however, requires a more delicate operation. There are several ways to define this product. We shall consider the so-called Cauchy’s product. Definition 5.2.5. Let Ý⬁ns0 a n and Ý⬁ns0 bn be two series in which the summation index starts at zero instead of one. Cauchy’s product of these two series is the series Ý⬁ns0 c n , where n

cn s

Ý ak bnyk ,

n s 0, 1, 2, . . . ,

ks0

that is, ⬁

Ý c n s a0 b0 q Ž a0 b1 q a1 b0 . q Ž a0 b2 q a1 b1 q a2 b0 . q ⭈⭈⭈ .

ns0

Other products could have been defined by simply adopting different arI rangements of the terms that make up the series Ý⬁ns0 c n . The question now is: under what condition will Cauchy’s product of two series converge? The answer to this question is given in the next theorem. Theorem 5.2.17. Let Ý⬁ns0 c n be Cauchy’s product of Ý⬁ns0 a n and Ý⬁ns0 bn . Suppose that these two series are convergent and have sums equal to s and t, respectively. 1. If at least one of Ý⬁ns0 a n and Ý⬁ns0 bn converges absolutely, then Ý⬁ns0 c n converges and its sum is equal to st Žthis result is known as Mertens’s theorem.. 2. If both series are absolutely convergent, then Ý⬁ns0 c n converges absolutely to the product st Žthis result is due to Cauchy.. Proof. 1. Suppose that Ý⬁ns0 a n is the series that converges absolutely. Let sn , t n , and u n denote the partial sums Ý nis0 a i , Ý nis0 bi , and Ý nis0 c i , respectively. We need to show that u n ™ st as n ™ ⬁. We have that u n s a0 b 0 q Ž a0 b1 q a1 b 0 . q ⭈⭈⭈ q Ž a0 bn q a1 bny1 q ⭈⭈⭈ qa n b 0 . s a0 t n q a1 t ny1 q ⭈⭈⭈ qan t 0 .

Ž 5.42 .

Let ␤n denote the remainder of the series Ý⬁ns0 bn with respect to t n , that is, ␤n s t y t n Ž n s 0, 1, 2, . . . .. By making the proper substitution in

164

INFINITE SEQUENCES AND SERIES

Ž5.42. we get u n s a0 Ž t y ␤n . q a1 Ž t y ␤ny1 . q ⭈⭈⭈ qan Ž t y ␤ 0 . s tsn y Ž a0 ␤n q a1 ␤ny1 q ⭈⭈⭈ qa n ␤ 0 . .

Ž 5.43 .

Since sn ™ s as n ™ ⬁, the proof of Ž1. will be complete if we can show that the sum inside parentheses in Ž5.43. goes to zero as n ™ ⬁. We now proceed to show that this is the case. Let ⑀ ) 0 be given. Since the sequence
␤n4⬁ns0 converges to zero, there exists an integer N such that

␤n - ⑀

if n ) N.

Hence, a0 ␤n q a1 ␤ny1 q ⭈⭈⭈ qa n ␤ 0 F a n ␤ 0 q a ny1 ␤ 1 q ⭈⭈⭈ qa nyN ␤N q a nyNy1 ␤Nq1 q a nyNy2 ␤Nq2 q ⭈⭈⭈ qa0 ␤n nyNy1

- a n ␤ 0 q a ny1 ␤ 1 q ⭈⭈⭈ qa nyN ␤N q ⑀

Ý

ai

is0 n

-B

Ý

Ž 5.44 .

a i q ⑀ s*,

isnyN

where B s max Ž< ␤ 0 < , < ␤ 1 < , . . . , < ␤N < . and s* is the sum of the series Ý⬁ns0 < a n < ŽÝ⬁ns0 a n is absolutely convergent.. Furthermore, because of this and by the Cauchy criterion we can find an integer M such that n

Ý

ai - ⑀

if n y N ) Mq 1.

isnyN

Thus when n ) N q Mq 1 we get from inequality Ž5.44. a0 ␤n q a1 ␤ny1 q ⭈⭈⭈ qa n ␤ 0 F ⑀ Ž B q s* . . Since ⑀ can be arbitrarily small, we conclude that lim Ž a0 ␤n q a1 ␤ny1 q ⭈⭈⭈ qa n ␤ 0 . s 0.

n™⬁

165

SEQUENCES AND SERIES OF FUNCTIONS

2. Let ®n denote the nth partial sum of Ý⬁is0 < c i < . Then ®n s a0 b 0 q a0 b1 q a1 b 0 q ⭈⭈⭈ q a0 bn q a1 bny1 q ⭈⭈⭈ qa n b 0 F a0 b 0 q a0 b1 q a1 b 0 q ⭈⭈⭈ q a0 bn q a1 bny1 q ⭈⭈⭈ q a n b 0 s a0 tUn q a1 tUny1 q ⭈⭈⭈ q a n tU0 , where tUk s Ý kis0 < bi < , k s 0, 1, 2, . . . , n. Thus, ®n F Ž < a0 < q < a1 < q ⭈⭈⭈ q < a n < . tUn F s*t*

for all n,

where t* is the sum of the series Ý⬁ns0 < bn < , which is convergent by assumption. We conclude that the sequence
®n4⬁ns0 is bounded. Since ®n G 0, then by Exercise 5.12 this sequence is convergent, and therefore Ý⬁ns0 c n converges absolutely. By part Ž1., the sum of this series is st. I It should be noted that absolute convergence of at least one of Ý⬁ns0 a n and Ý⬁ns0 bn is an essential condition for the validity of part Ž1. of Theorem 5.2.17. If this condition is not satisfied, then Ý⬁ns0 c n may not converge. For example, consider the series Ý⬁ns0 a n , Ý⬁ns0 bn , where

a n s bn s

Ž y1.

n

'n q 1

,

n s 0, 1, . . . .

These two series are convergent by Theorem 5.2.14. They are not, however, absolutely convergent, and their Cauchy’s product is divergent Žsee Exercise 5.22..

5.3. SEQUENCES AND SERIES OF FUNCTIONS All the sequences and series considered thus far in this chapter had constant terms. We now extend our study to sequences and series whose terms are functions of x. Definition 5.3.1. set D ; R.

Let
f nŽ x .4⬁ns1 be a sequence of functions defined on a

166

INFINITE SEQUENCES AND SERIES

1. If there exists a function f Ž x . defined on D such that for every x in D, lim f n Ž x . s f Ž x . ,

n™⬁

then the sequence
f nŽ x .4⬁ns1 is said to converge to f Ž x . on D. Thus for a given ⑀ ) 0 there exists an integer N such that < f nŽ x . y f Ž x .< - ⑀ if n ) N. In general, N depends on ⑀ as well as on x. 2. If Ý⬁ns1 f nŽ x . converges for every x in D to sŽ x ., then sŽ x . is said to be the sum of the series. In this case, for a given ⑀ ) 0 there exists an integer N such that sn Ž x . y s Ž x . - ⑀

if n ) N,

where snŽ x . is the nth partial sum of the series Ý⬁ns1 f nŽ x .. The integer N depends on ⑀ and, in general, on x also. 3. In particular, if N in Ž1. depends on ⑀ but not on x g D, then the sequence
f nŽ x .4⬁ns1 is said to converge uniformly to f Ž x . on D. Similarly, if N in Ž2. depends on ⑀ , but not on x g D, then the series Ý⬁ns1 f nŽ x . converges uniformly to sŽ x . on D. I The Cauchy criterion for sequences ŽTheorem 5.1.6. and its application to series ŽTheorem 5.2.1. apply to sequences and series of functions. In case of uniform convergence, the integer N described in this criterion depends only on ⑀ . Theorem 5.3.1. Let
f nŽ x .4⬁ns1 be a sequence of functions defined on D ; R and converging to f Ž x .. Define the number ␭ n as

␭ n s sup f n Ž x . y f Ž x . . xgD

Then the sequence converges uniformly to f Ž x . on D if and only if ␭ n ™ 0 as n ™ ⬁. Proof. Sufficiency: Suppose that ␭ n ™ 0 as n ™ ⬁. To show that f nŽ x . ™ f Ž x . uniformly on D. Let ⑀ ) 0 be given. Then there exists an integer N such that for n ) N, ␭ n - ⑀ . Hence, for such values of n, fn Ž x . y f Ž x . F ␭n - ⑀ for all xg D. Since N depends only on ⑀ , the sequence
f nŽ x .4⬁ns1 converges uniformly to f Ž x . on D. Necessity: Suppose that f nŽ x . ™ f Ž x . uniformly on D. To show that ␭ n ™ 0. Let ⑀ ) 0 be given. There exists an integer N that depends only on ⑀ such that for n ) N, ⑀ fn Ž x . y f Ž x . 2

167

SEQUENCES AND SERIES OF FUNCTIONS

for all xg D. It follows that

⑀

␭ n s sup f n Ž x . y f Ž x . F

2

xgD

Thus ␭ n ™ 0 as n ™ ⬁.

.

I

Theorem 5.3.1 can be applied to convergent series of functions by replacing f nŽ x . and f Ž x . with snŽ x . and sŽ x ., respectively, where snŽ x . is the nth partial sum of the series and sŽ x . is its sum. EXAMPLE 5.3.1. Let f nŽ x . s sinŽ2␲ xrn., 0 F xF 1. Then f nŽ x . ™ 0 as n ™ ⬁. Furthermore, sin

ž / 2␲ x n

F

2␲ x n

F

2␲ n

.

In this case,

␭ n s sup

sin

0FxF1

ž / 2␲ x n

F

2␲ n

.

Thus ␭ n ™ 0 as n ™ ⬁, and the sequence
f nŽ x .4⬁ns1 converges uniformly to f Ž x . s 0 on w0, 1x. The next theorem provides a simple test for uniform convergence of series of functions. It is due to Karl Weierstrass Ž1815᎐1897.. Theorem 5.3.2 ŽWeierstrass’s M-test.. Let Ý⬁ns1 f nŽ x . be a series of functions defined on D ; R. If there exists a sequence
Mn4⬁ns1 of constants such that f n Ž x . F Mn ,

n s 1, 2,, . . . ,

for all xg D, and if Ý⬁ns1 Mn converges, then Ý⬁ns1 f nŽ x . converges uniformly on D. Proof. Let ⑀ ) 0 be given. By the Cauchy criterion ŽTheorem 5.2.1., there exists an integer N such that n

Ý

ismq1

Mi - ⑀

168

INFINITE SEQUENCES AND SERIES

for all n ) m ) N. Hence, for all such values of m, n, and for all x g D, n

Ý

n

Ý

fi Ž x . F

ismq1

fi Ž x .

ismq1 n

Ý

F

Mi - ⑀ .

ismq1

This implies that Ý⬁ns1 f nŽ x . converges uniformly on D by the Cauchy criterion. I We note that Weierstrass’s M-test is easier to apply than Theorem 5.3.1, since it does not require specifying the sum sŽ x . of the series. EXAMPLE 5.3.2. Let us investigate convergence of the sequence
f nŽ x .4⬁ns1 , where f nŽ x . is defined as

°2 xq 1 , ~

f n Ž x . s exp

n x

ž /

n 1 1y , n

¢

0 F x- 1, 1 F x- 2,

,

xG 2.

This sequence converges to f Ž x. s

½

2 x, 1,

0 F x- 1, xG 1.

Now, fn Ž x . y f Ž x .

°1rn, ¢1rn,

0 F x - 1, 1 F x- 2, xG 2.

s~exp Ž xrn . y 1,

However, for 1 F x- 2, exp Ž xrn . y 1 - exp Ž 2rn . y 1. Furthermore, by Maclaurin’s series expansion, exp

ž / 2

n

y1s

⬁

Ý

ks1

Ž 2rn . k!

k

)

2 n

)

1 n

.

169

SEQUENCES AND SERIES OF FUNCTIONS

Thus, sup

f n Ž x . y f Ž x . s exp Ž 2rn . y 1,

0Fx-⬁

which tends to zero as n ™ ⬁. Therefore, the sequence
f nŽ x .4⬁ns1 converges uniformly to f Ž x . on w0, ⬁.. EXAMPLE 5.3.3.

Consider the series Ý⬁ns1 f nŽ x ., where fn Ž x . s

xn n3 q nx n

0 F x F 1.

,

The function f nŽ x . is monotone increasing with respect to x. It follows that for 0 F xF 1, fn Ž x . s fn Ž x . F

1 n qn 3

,

n s 1, 2, . . . .

But the series Ý⬁ns1 w1rŽ n3 q n.x is convergent. Hence, Ý⬁ns1 f nŽ x . is uniformly convergent on w0, 1x by Weierstrass’s M-test. 5.3.1. Properties of Uniformly Convergent Sequences and Series Sequences and series of functions that are uniformly convergent have several interesting properties. We shall study some of these properties in this section. Theorem 5.3.3. Let
f nŽ x .4⬁ns1 be uniformly convergent to f Ž x . on a set D. If for each n, f nŽ x . has a limit ␶n as x™ x 0 , where x 0 is a limit point of D, then the sequence
␶n4⬁ns1 converges to ␶ 0 s lim x ™ x 0 f Ž x .. This is equivalent to stating that lim

n™⬁

lim f n Ž x . s lim

x™x 0

x™x 0

lim f n Ž x . .

n™⬁

Proof. Let us first show that
␶n4⬁ns1 is a convergent sequence. By the Cauchy criterion ŽTheorem 5.1.6., there exists an integer N such that for a given ⑀ ) 0, fm Ž x . y fn Ž x . -

⑀ 2

for all m ) N, n ) N.

Ž 5.45 .

The integer N depends only on ⑀ , and inequality Ž5.45. is true for all x g D, since the sequence is uniformly convergent. By taking the limit as x™ x 0 in Ž5.45. we get < ␶m y ␶n < F

⑀ 2

if m ) N, n ) N,

170

INFINITE SEQUENCES AND SERIES

which indicates that
␶n4⬁ns1 is a Cauchy sequence and is therefore convergent. Let ␶ 0 s lim n™⬁ ␶n . We now need to show that f Ž x . has a limit and that this limit is equal to ␶ 0 . Let ⑀ ) 0 be given. There exists an integer N1 such that for n ) N1 , f Ž x . y fn Ž x . -

⑀ 4

for all xg D, by the uniform convergence of the sequence. Furthermore, there exists an integer N2 such that < ␶n y ␶ 0 < -

⑀

if n ) N2 .

4

Thus for n ) maxŽ N1 , N2 ., f Ž x . y f n Ž x . q < ␶n y ␶ 0 < -

⑀ 2

for all xg D. Then f Ž x . y ␶ 0 F f Ž x . y f n Ž x . q f n Ž x . y ␶n q ␶ n y ␶ 0 - f n Ž x . y ␶n q

⑀

Ž 5.46 .

2

if n ) maxŽ N1 , N2 . for all xg D. By taking the limit as x™ x 0 in Ž5.46. we get lim f Ž x . y ␶ 0 F

x™x 0

⑀

Ž 5.47 .

2

by the fact that lim f n Ž x . y ␶n s 0

x™x 0

for n s 1, 2, . . . .

Since ⑀ is arbitrarily small, inequality Ž5.47. implies that lim f Ž x . s ␶ 0 .

x™x 0

I

Corollary 5.3.1. Let
f nŽ x .4⬁ns1 be a sequence of continuous functions that converges uniformly to f Ž x . on a set D. Then f Ž x . is continuous on D. Proof. The proof follows directly from Theorem 5.3.3, since ␶n s f nŽ x 0 . for I n G 1 and ␶ 0 s lim n™⬁ ␶n s lim n™⬁ f nŽ x 0 . s f Ž x 0 ..

171

SEQUENCES AND SERIES OF FUNCTIONS

Corollary 5.3.2. Let Ý⬁ns1 f nŽ x . be a series of functions that converges uniformly to sŽ x . on a set D. If for each n, f nŽ x . has a limit ␶n as x™ x 0 , then the series Ý⬁ns1 ␶n converges and has a sum equal to s0 s lim x ™ x 0 sŽ x ., that is, ⬁

lim

⬁

Ý fn Ž x . s Ý

x™x 0 ns1

lim f n Ž x . .

ns1 x™x 0

I

Proof. The proof is left to the reader.

By combining Corollaries 5.3.1 and 5.3.2 we conclude the following corollary: Corollary 5.3.3. Let Ý⬁ns1 f nŽ x . be a series of continuous functions that converges uniformly to sŽ x . on a set D. Then sŽ x . is continuous on D. EXAMPLE 5.3.4. Let f nŽ x . s x 2rŽ1 q x 2 . ny1 be defined on w1, ⬁. for n G 1. Let snŽ x . be the nth partial sum of the series Ý⬁ns1 f nŽ x .. Then,

sn Ž x . s

n

Ý

ks1

x

1y

2

Ž1qx2 .

ky1

sx

2

1

Ž1qx2 .

1y

1

n

,

1qx2

by using the fact that the sum of the finite geometric series Ý nks1 a ky1 is n

Ý a ky1 s

ks1

1 y an 1ya

.

Since 1rŽ1 q x 2 . - 1 for xG 1, then as n ™ ⬁, x2

sn Ž x . ™ 1y

s1qx2.

1 1qx2

Thus, ⬁

Ý

ns1

x2

Ž1qx . 2

ny1

s1qx2.

Ž 5.48 .

172

INFINITE SEQUENCES AND SERIES

Now, let x 0 s 1, then ⬁

Ý

lim

ns1 x™1

x2

Ž1qx2 .

ny1

s

⬁

Ý

ns1

1 2

ny1

s

1 1 y 12

s2

s lim Ž 1 q x 2 . , x™1

which results from applying formula Ž5.48. with as 12 and then letting n ™ ⬁. This provides a verification to Corollary 5.3.2. Note that the series Ý⬁ns1 f nŽ x . is uniformly convergent by Weierstrass’s M-test Žwhy?.. Corollaries 5.3.2 and 5.3.3 clearly show that the properties of the function f nŽ x . carry over to the sum sŽ x . of the series Ý⬁ns1 f nŽ x . when the series is uniformly convergent. Another property that sŽ x . shares with the f nŽ x .’s is given by the following theorem: Theorem 5.3.4. Let Ý⬁ns1 f nŽ x . be a series of functions, where f nŽ x . is differentiable on w a, b x for n G 1. Suppose that Ý⬁ns1 f nŽ x . converges at least at one point x 0 g w a, b x and that Ý⬁ns1 f nX Ž x . converges uniformly on w a, b x. Then we have the following: 1. Ý⬁ns1 f nŽ x . converges uniformly to sŽ x . on w a, b x. 2. s⬘Ž x . s Ý⬁ns1 f nX Ž x ., that is, the derivative of sŽ x . is obtained by a term-by-term differentiation of the series Ý⬁ns1 f nŽ x .. Proof. 1. Let x/ x 0 be a point in w a, b x. By the mean value theorem ŽTheorem 4.2.2., there exists a point ␰ n between x and x 0 such that for n G 1, f n Ž x . y f n Ž x 0 . s Ž xy x 0 . f nX Ž ␰ n . .

Ž 5.49 .

Since Ý⬁ns1 f nX Ž x . is uniformly convergent on w a, b x, then by the Cauchy criterion, there exists an integer N such that n

Ý

f iX Ž x . -

ismq1

⑀ by a

for all n ) m ) N and for any xg w a, b x. From Ž5.49. we get n

Ý

f i Ž x . y f i Ž x 0 . s < xy x 0 <

ismq1

n

Ý

ismq1

-

⑀

by a -⑀

< xy x 0 <

f iX Ž ␰ i .

173

SEQUENCES AND SERIES OF FUNCTIONS

for all n ) m ) N and for any xg w a, b x. This shows that ⬁

Ý

fn Ž x . y fn Ž x 0 .

ns1

is uniformly convergent on D. Consequently, ⬁

⬁

ns1

ns1

Ý fn Ž x . s Ý

fn Ž x . y fn Ž x 0 . q s Ž x 0 .

is uniformly convergent to sŽ x . on D, where sŽ x 0 . is the sum of the series Ý⬁ns1 f nŽ x 0 ., which was assumed to be convergent. 2. Let ␾nŽ h. denote the ratio f n Ž xq h . y f n Ž x .

␾n Ž h . s

h

n s 1, 2, . . . ,

,

where both x and xq h belong to w a, b x. By invoking the mean value theorem again, ␾nŽ h. can be written as

␾n Ž h . s f nX Ž xq ␪n h . ,

n s 1, 2, . . . ,

where 0 - ␪n - 1. Furthermore, by the uniform convergence of Ý⬁ns1 f nX Ž x . we can deduce that Ý⬁ns1 ␾nŽ h. is also uniformly convergent on wyr, r x for some r ) 0. But ⬁

Ý

␾n Ž h . s

ns1

⬁

Ý

f n Ž xq h . y f n Ž x . h

ns1

s

s Ž x q h. y s Ž x . h

,

Ž 5.50 .

where sŽ x . is the sum of the series Ý⬁ns1 f nŽ x .. Let us now apply Corollary 5.3.2 to Ý⬁ns1 ␾nŽ h.. We get ⬁

lim

Ý

h™0 ns1

␾n Ž h . s

⬁

Ý

lim ␾n Ž h . .

ns1 h™0

From Ž5.50. and Ž5.51. we then have lim

s Ž x q h. y s Ž x .

h™0

h

s

⬁

Ý f nX Ž x . .

ns1

Thus, s⬘ Ž x . s

⬁

Ý f nX Ž x . .

ns1

I

Ž 5.51 .

174

INFINITE SEQUENCES AND SERIES

5.4. POWER SERIES A power series is a special case of the series of functions discussed in Section 5.3. It is of the form Ý⬁ns0 a n x n, where the a n’s are constants. We have already encountered such series in connection with Taylor’s and Maclaurin’s series in Section 4.3. Obviously, just as with any series of functions, the convergence of a power series depends on the values of x. By definition, if there exists a number ␳ ) 0 such that Ý⬁ns0 a n x n is convergent if < x < - ␳ and is divergent if < x < ) ␳ , then ␳ is said to be the radius of convergence of the series, and the interval Žy␳ , ␳ . is called the interval of convergence. The set of all values of x for which the power series converges is called its region of convergence. The definition of the radius of convergence implies that Ý⬁ns0 a n x n is absolutely convergent within its interval of convergence. This is shown in the next theorem. Theorem 5.4.1. Let ␳ be the radius of convergence of Ý⬁ns0 a n x n. Suppose that ␳ ) 0. Then Ý⬁ns0 a n x n converges absolutely for all x inside the interval Žy␳ , ␳ .. Proof. Let x be such that < x < - ␳ . There exists a point x 0 g Žy␳ , ␳ . such that < x < - < x 0 < . Then, Ý⬁ns0 a n x 0n is a convergent series. By Result 5.2.1, a n x 0n ™ 0 as n ™ ⬁, and hence
a n x 0n4⬁ns0 is a bounded sequence by Theorem 5.1.1. Thus < a n x 0n < - K for all n. Now, < an x n < s an

ž / x

x0

n

x 0n

- K␩ n , where

␩s

x x0

- 1.

Since the geometric series Ý⬁ns0 ␩ n is convergent, then by the comparison test Žsee Theorem 5.2.4., the series Ý⬁ns0 < a n x n < is convergent. I To determine the radius of convergence we shall rely on some of the tests of convergence given in Section 5.2.1. Theorem 5.4.2.

Let Ý⬁ns0 a n x n be a power series. Suppose that lim

n™⬁

a nq1 an

s p.

175

POWER SERIES

Then the radius of convergence of the power series is

° ¢

1rp, ␳ s 0, ⬁,

~

0 - p- ⬁, ps ⬁, ps 0.

Proof. The proof follows from applying the ratio test given in Theorem 5.2.6 to the series Ý⬁ns0 < a n x n < : We have that if lim

a nq1 x nq1 an x n

n™⬁

- 1,

then Ý⬁ns0 a n x n is absolutely convergent. This inequality can be written as p < x < - 1.

Ž 5.52 .

If 0 - p- ⬁, then absolute convergence occurs if < x < - 1rp and the series diverges when < x < ) 1rp. Thus ␳ s 1rp. If ps ⬁, the series diverges whenever x/ 0. In this case, ␳ s 0. If ps 0, then Ž5.52. holds for any value of x, that is, ␳ s ⬁. I Theorem 5.4.3.

Let Ý⬁ns0 a n x n be a power series. Suppose that lim sup < a n < 1r n s q. n™⬁

Then,

°1rq, ¢⬁,

␳ s~0,

0 - q - ⬁, q s ⬁, q s 0.

Proof. This result follows from applying the root test in Theorem 5.2.8 to the series Ý⬁ns0 < a n x n < . Details of the proof are similar to those given in Theorem 5.4.2. I The determination of the region of convergence of Ý⬁ns0 a n x n depends on the value of ␳ . We know that the series converges if < x < - ␳ and diverges if < x < ) ␳ . The convergence of the series at xs ␳ and x s y␳ has to be determined separately. Thus the region of convergence can be Žy␳ , ␳ ., wy␳ , ␳ ., Žy␳ , ␳ x, or wy␳ , ␳ x. EXAMPLE 5.4.1. Consider the geometric series Ý⬁ns0 x n. By applying either Theorem 5.4.2 or Theorem 5.4.3, it is easy to show that ␳ s 1. The series diverges if xs 1 or y1. Thus the region of convergence is Žy1, 1.. The sum

176

INFINITE SEQUENCES AND SERIES

of this series can be obtained from formula Ž5.48. by letting n go to infinity. Thus ⬁

Ý

xns

ns0

1 1yx

y1 - x- 1.

,

Ž 5.53 .

EXAMPLE 5.4.2. Consider the series Ý⬁ns0 Ž x nrn!.. Here, lim

a nq1 an

n™⬁

s lim

n™⬁

s lim

n™⬁

n!

Ž n q 1. ! 1 nq1

s 0.

Thus ␳ s ⬁, and the series converges absolutely for any value of x. This particular series is Maclaurin’s expansion of e x , that is, exs

⬁

xn

ns0

n!

Ý

.

EXAMPLE 5.4.3. Suppose we have the series Ý⬁ns1 Ž x nrn.. Then lim

a nq1

n™⬁

an

s lim

n™⬁

n nq1

s 1,

and ␳ s 1. When xs 1 we get the harmonic series, which is divergent. When x s y1 we get the alternating harmonic series, which is convergent by Theorem 5.2.14. Thus the region of convergence is wy1, 1.. In addition to being absolutely convergent within its interval of convergence, a power series is also uniformly convergent there. This is shown in the next theorem. Theorem 5.4.4. Let Ý⬁ns0 a n x n be a power series with a radius of convergence ␳ Ž) 0.. Then we have the following: 1. The series converges uniformly on the interval wyr, r x, where r - ␳ . 2. If sŽ x . s Ý⬁ns0 a n x n, then sŽ x . Ži. is continuous on wyr, r x; Žii. is differentiable on wyr, r x and has derivative s⬘ Ž x . s

⬁

Ý nan x ny1 ,

ns1

yr F xF r ;

177

POWER SERIES

and Žiii. has derivatives of all orders on wyr, r x and dksŽ x . dx

k

s

⬁

a n n!

x nyk , Ý nsk Ž n y k . !

k s 1, 2, . . . , yr F xF r .

Proof. 1. If < x < F r, then < a n x n < F < a n < r n for n G 0. Since Ý⬁ns0 < a n < r n is convergent by Theorem 5.4.1, then by the Weierstrass M-test ŽTheorem 5.3.2., Ý⬁ns0 a n x n is uniformly convergent on wyr, r x. 2. Ži. Continuity of sŽ x . follows directly from Corollary 5.3.3. Žii. To show this result, we first note that the two series Ý⬁ns0 a n x n and Ý⬁ns1 nan x ny1 have the same radius of convergence. This is true by Theorem 5.4.3 and the fact that lim sup < nan < 1r n s lim sup < a n < 1r n , n™⬁

n™⬁

since lim n™⬁ n1r n s 1 as n ™ ⬁. We can then assert that Ý⬁ns1 nan x ny1 is uniformly convergent on wyr, r x. By Theorem 5.3.4, sŽ x . is differentiable on wyr, r x, and its derivative is obtained by a term-by-term differentiation of Ý⬁ns0 a n x n. Žiii. This follows from part Žii. by repeated differentiation of sŽ x .. I Under a certain condition, the interval on which the power series converges uniformly can include the end points of the interval of convergence. This is discussed in the next theorem. Theorem 5.4.5. Let Ý⬁ns0 a n x n be a power series with a finite nonzero radius of convergence ␳ . If Ý⬁ns0 a n ␳ n is absolutely convergent, then the power series is uniformly convergent on wy␳ , ␳ x. Proof. The proof is similar to that of part 1 of Theorem 5.4.4. In this case, for < x < F ␳ , < a n x n < F < a n < ␳ n. Since Ý⬁ns0 < a n < ␳ n is convergent, then Ý⬁ns0 a n x n I is uniformly convergent on wy␳ , ␳ x by the Weierstrass M-test. EXAMPLE 5.4.4. Consider the geometric series of Example 5.4.1. This series is uniformly convergent on wyr, r x, where r - 1. Furthermore, by differentiating the two sides of Ž5.53. we get ⬁

1

ns1

Ž1yx.

Ý nx ny1 s

2

,

y1 - x - 1.

178

INFINITE SEQUENCES AND SERIES

This provides a series expansion of 1rŽ1 y x . 2 within the interval Žy1, 1.. By repeated differentiation it is easy to show that for y1 - x- 1, 1

Ž1yx.

k

s

⬁

Ý

ns0

ž

/

nqky1 n x , n

k s 1, 2, . . . .

The radius of convergence of this series is ␳ s 1, the same as for the original series. EXAMPLE 5.4.5. Suppose we have the series ⬁

Ý

ns1

ž

2n

x

2 n2 q n 1 y x

/

n

,

which can be written as ⬁

zn

ns1

2 n2 q n

Ý

,

where zs 2 xrŽ1 y x .. This is a power series in z. By Theorem 5.4.2, the radius of convergence of this series is ␳ s 1. We note that when z s 1 the series Ý⬁ns1 w1rŽ2 n2 q n.x is absolutely convergent. Thus by Theorem 5.4.5, the given series is uniformly convergent for < z < F 1, that is, for values of x satisfying y

1 2

F

x 1yx

F

1 2

,

or equivalently, y1 F xF 13 .

5.5. SEQUENCES AND SERIES OF MATRICES In Section 5.3 we considered sequences and series whose terms were scalar functions of x rather than being constant as was done in Sections 5.1 and 5.2. In this section we consider yet another extension, in which the terms of the series are matrices rather than scalars. We shall provide a brief discussion of this extension. The interested reader can find a more detailed study of this topic in Gantmacher Ž1959., Lancaster Ž1969., and Graybill Ž1983.. As in Chapter 2, all matrix elements considered here are real. For the purpose of our study of sequences and series of matrices we first need to introduce the norm of a matrix.

179

SEQUENCES AND SERIES OF MATRICES

Definition 5.5.1. Let A be a matrix of order m = n. A norm of A, denoted by 5 A 5, is a real-valued function of A with the following properties: 1. 2. 3. 4.

5 A 5 G 0, and 5 A 5 s 0 if and only if A s 0. 5 cA 5 s < c < 5 A 5, where c is a scalar. 5 A q B 5 F 5 A 5 q 5 B 5, where B is any matrix of order m = n. 5 AC 5 F 5 A 5 5 C 5, where C is any matrix for which the product AC is defined. I

If A s Ž a i j ., then examples of matrix norms that satisfy properties 1, 2, 3, and 4 include the following: n 2 .1r2 1. The Euclidean norm, 5 A 5 2 s ŽÝ m . is1 Ý js1 a i j 1r2 2. The spectral norm, 5 A 5 s s w emax ŽA⬘A.x , where emax ŽA⬘A. is the largest eigenvalue of A⬘A.

Definition 5.5.2. Let A k s Ž a i jk . be matrices of orders m = n for k G 1. The sequence
A k 4⬁ks1 is said to converge to the m = n matrix A s Ž a i j . if lim k ™⬁ a i jk s a i j for i s 1, 2, . . . , m; j s 1, 2, . . . , n. I For example, the sequence of matrices 1 Aks

k

1 k

2

2

ky1

y1

kq1 k2

0

,

k s 1, 2,, . . . ,

2yk2

converges to As

0 2

y1 0

1 y1

as k ™ ⬁. The sequence 1 Aks

k 1

k2y2 k

,

k s 1, 2, . . . ,

1qk

does not converge, since k 2 y 2 goes to infinite as k ™ ⬁.

180

INFINITE SEQUENCES AND SERIES

From Definition 5.5.2 it is easy to see that
A k 4⬁ks1 converges to A if and only if lim 5 A k y A 5 s 0,

k™⬁

where 5 ⭈ 5 is any matrix norm. Definition 5.5.3. Let
A k 4⬁ks1 be a sequence of matrices of order m = n. Then Ý⬁ks1 A k is called an infinite series Žor just a series. of matrices. This series is said to converge to the m = n matrix Ss Ž si j . if and only if the series Ý⬁ks1 a i jk converges for all i s 1, 2, . . . , m; j s 1, 2, . . . , n, where a i jk is the Ž i, j .th element of A k , and ⬁

Ý ai jk s si j ,

i s 1, 2, . . . , m; j s 1, 2, . . . , n.

Ž 5.54 .

ks1

The series Ý⬁ks1 A k is divergent if at least one of the series in Ž5.54. is divergent. I From Definition 5.5.3 and Result 5.2.1 we conclude that Ý⬁ks1 A k diverges if lim k ™⬁ a i jk / 0 for at least one pair Ž i, j ., that is, if lim k ™⬁ A k / 0. A particular type of infinite series of matrices is the power series Ý⬁ks0 ␣ k Ak , where A is a square matrix, ␣ k is a scalar Ž k s 0, 1, . . . ., and A0 is by definition the identity matrix I. For example, the power series IqAq

1 2!

A2 q

1 3!

A3 q ⭈⭈⭈ q

1 k!

Ak q ⭈⭈⭈

represents an expansion of the exponential matrix function expŽA. Žsee Gantmacher, 1959.. Theorem 5.5.1. Let A be an n = n matrix. Then lim k ™⬁ Ak s 0 if 5 A 5 - 1, where 5 ⭈ 5 is any matrix norm. Proof. From property 4 in Definition 5.5.1 we can write 5 Ak 5 F 5 A 5 k ,

k s 1, 2, . . . .

Since 5 A 5 - 1, then lim k ™⬁ 5 Ak 5 s 0, which implies that lim k ™⬁ Ak s 0 Žwhy?.. I Theorem 5.5.2. Let A be a symmetric matrix of order n = n such that < ␭ i < - 1 for i s 1, 2, . . . , n, where ␭ i is the ith eigenvalue of A Žall the eigenvalues of A are real by Theorem 2.3.5.. Then Ý⬁ks0 Ak converges to ŽI y A.y1 .

181

SEQUENCES AND SERIES OF MATRICES

Proof. By the spectral decomposition theorem ŽTheorem 2.3.10. there exists an orthogonal matrix P such that A s P⌳ P⬘, where ⌳ is a diagonal matrix whose diagonal elements are the eigenvalues of A. Then Ak s P⌳k P⬘,

k s 0, 1, 2, . . . .

Since < ␭ i < - 1 for all i, then ⌳k ™ 0 and hence Ak ™ 0 as k ™ ⬁. Furthermore, the matrix I y A is nonsingular, since I y A s P Ž I y ⌳ . P⬘ and all the diagonal elements of I y ⌳ are positive. Now, for any nonnegative integer k we have the following identity:

Ž I y A . Ž I q A q A2 q ⭈⭈⭈ qAk . s I y Akq1 . Hence, I q A q ⭈⭈⭈ qAk s Ž I y A .

y1

Ž I y Akq1 . .

By letting k go to infinity we get ⬁

Ý Ak s Ž I y A. y1 ,

ks0

since lim k ™⬁ Akq1 s 0.

I

Theorem 5.5.3. Let A be a symmetric n = n matrix and ␭ be any eigenvalue of A. Then < ␭ < F 5 A 5, where 5 A 5 is any matrix norm of A. Proof. We have that Av s ␭v, where v is an eigenvector of A for the eigenvalue ␭. If 5 A 5 is any matrix norm of A, then 5 ␭v 5 s < ␭ < 5 v 5 s 5 Av 5 F 5 A 5 5 v 5 . Since v / 0, we conclude that < ␭ < F 5 A 5.

I

Corollary 5.5.1. Let A be a symmetric matrix of order n = n such that 5 A 5 - 1, where 5 A 5 is any matrix norm of A. Then Ý⬁ks0 Ak converges to ŽI y A.y1 . Proof. This result follows from Theorem 5.5.2, since for i s 1, 2, . . . , n, < ␭ i < F 5 A 5 - 1. I

182

INFINITE SEQUENCES AND SERIES

5.6. APPLICATIONS IN STATISTICS Sequences and series have many useful applications in statistics. Some of these applications will be discussed in this section. 5.6.1. Moments of a Discrete Distribution Perhaps one of the most visible applications of infinite series in statistics is in the study of the distribution of a discrete random variable that can assume a countable number of values. Under certain conditions, this distribution can be completely determined by its moments. By definition, the moments of a distribution are a set of descriptive constants that are useful for measuring its properties. Let X be a discrete random variable that takes on the values x 0 , x 1 , . . . , x n , . . . , with probabilities pŽ n., n G 0. Then, by definition, the kth central moment of X, denoted by ␮ k , is

␮k s E Ž X y ␮ .

k

⬁

s

Ý Ž x n y ␮ . k p Ž n. ,

k s 1, 2, . . . ,

ns0

where ␮ s E Ž X . s Ý⬁ns0 x n pŽ n. is the mean of X. We note that ␮ 2 s ␴ 2 is the variance of X. The kth noncentral moment of X is given by the series

␮Xk s E Ž X k . s

⬁

Ý

x nk p Ž n . ,

k s 1, 2, . . . .

Ž 5.55 .

ns0

We note that ␮X1 s ␮. If, for some integer N, < x n < G 1 for n ) N, and if the series in Ž5.55. converges absolutely, then so does the series for ␮Xj Ž j s 1, 2, . . . , k y 1.. This follows from applying the comparison test: < x n < j p Ž n. F < x n < k pŽ n.

if j - k and n ) N.

Examples of discrete random variables with a countable number of values include the Poisson Žsee Section 4.5.3. and the negative binomial. The latter random variable represents the number n of failures before the r th success when independent trials are performed, each of which has two probability outcomes, success or failure, with a constant probability p of success on each trial. Its probability mass function is therefore of the form p Ž n. s

ž

/

n nqry1 r p Ž1 y p. , n

n s 0, 1, 2, . . . .

183

APPLICATIONS IN STATISTICS

By contrast, the Poisson random variable has the probability mass function ey␭␭ n

p Ž n. s

n!

n s 0, 1, 2, . . . ,

,

where ␭ is the mean of X. We can verify that ␭ is the mean by writing

␮s

⬁

Ýn

ey␭␭ n n!

ns0

y␭

s ␭e

␭ ny1

⬁

Ý Ž n y 1. !

ns1

s ␭ ey␭

␭n

⬁

Ý

ns0

n!

s ␭ ey␭ Ž e ␭ . ,

by Maclaurin’s expansion of e ␭

s ␭. The second noncentral moment of the Poisson distribution is

␮X2 s

⬁

Ý

n2

ey␭␭ n n!

ns0

s ey␭␭

␭ ny1

⬁

Ý n Ž n y 1. !

ns1

s ey␭␭

⬁

Ý

Ž n y 1 q 1.

ns1

s ey␭␭ ␭

⬁

␭ ny2

␭ ny1

Ž n y 1. ! ⬁

␭ ny1

q Ý Ý ns2 Ž n y 2 . ! ns1 Ž n y 1 . !

s ey␭␭w ␭ e ␭ q e ␭ x s ␭2 q ␭ . In general, the kth noncentral moment of the Poisson distribution is given by the series

␮Xk s

⬁

Ý nk

ns0

ey␭␭ n n!

,

k s 1, 2, . . . ,

184

INFINITE SEQUENCES AND SERIES

which converges for any k. This can be shown, for example, by the ratio test

lim

a nq1

s lim

an

n™⬁

n™⬁

ž

nq1 n

/

␭

k

nq1

s 0 - 1. Thus all the noncentral moments of the Poisson distribution exist. Similarly, for the negative binomial distribution we have ⬁

Ýn

␮s

ns0

ž

/

n nqry1 r p Ž1 y p. n

r Ž1 y p.

s

Ž why? . ,

p ⬁

␮X2 s

Ý n2

ns0

s

ž

Ž 5.56 .

/

n nqry1 r p Ž1 y p. n

r Ž 1 y p . Ž 1 q r y rp .

Ž why? .

p2

Ž 5.57 .

and the kth noncentral moment,

␮Xk s

⬁

Ý nk

ns0

ž

/

n nqry1 r p Ž1 y p. , n

k s 1, 2, . . . ,

Ž 5.58 .

exists for any k, since, by the ratio test,

lim

n™⬁

a nq1 an

s lim

n™⬁

ž

nq1

/ž / k

nqr nq1 n Ž1 y p. nqry1 n

ž

s Ž 1 y p . lim

n™⬁

/

ž

nq1 n

/ž k

nqr nq1

/

s 1 y p- 1, which proves convergence of the series in Ž5.58.. A very important inequality that concerns the mean ␮ and variance ␴ 2 of any random variable X Žnot just the discrete ones. is Chebyshev’s inequality,

185

APPLICATIONS IN STATISTICS

namely, P Ž < X y ␮ < G r␴ . F

1 r2

,

or equivalently, P Ž < X y ␮ < - r␴ . G 1 y

1 r2

Ž 5.59 .

,

where r is any positive number Žsee, for example, Lindgren, 1976, Section 2.3.2.. The importance of this inequality stems from the fact that it is independent of the exact distribution of X and connects the variance of X with the distribution of its values. For example, inequality Ž5.59. states that at least Ž1 y 1rr 2 . = 100% of the values of X fall within r␴ from its mean, where ␴ s '␴ 2 is the standard deviation of X. Chebyshev’s inequality is a special case of a more general inequality called Markov’s inequality. If b is a nonzero constant and hŽ x . is a nonnegative function, then 1

P hŽ X . G b2 F

b2

E hŽ X . ,

provided that E w hŽ X .x exists. Chebyshev’s inequality follows from Markov’s inequality by choosing hŽ X . s Ž X y ␮ . 2 . Another important result that concerns the moments of a distribution is given by the following theorem, regarding what is known as the Stieltjes moment problem, which also applies to any random variable: Theorem 5.6.1. Suppose that the moments ␮Xk Ž k s 1, 2, . . . . of a random variable X exist, and the series ⬁

Ý

ks1

␮Xk k!

Ž 5.60 .

␶k

is absolutely convergent for some ␶ ) 0. Then these moments uniquely determine the cumulative distribution function F Ž x . of X. Proof. See, for example, Fisz Ž1963, Theorem 3.2.1..

I

In particular, if < ␮Xk < F M k ,

k s 1, 2, . . . ,

for some constant M, then the series in Ž5.60. converges absolutely for any ␶ ) 0 by the comparison test. This is true because the series Ý⬁ks1 Ž M krk!.␶ k converges Žfor example, by the ratio test. for any value of ␶ .

186

INFINITE SEQUENCES AND SERIES

It should be noted that absolute convergence of the series in Ž5.60. is a sufficient condition for the unique determination of F Ž x ., but is not a necessary condition. This is shown in Rao Ž1973, page 106.. Furthermore, if some moments of X fail to exist, then the remaining moments that do exist cannot determine F Ž x . uniquely. The following counterexample is given in Fisz Ž1963, page 74.: Let X be a discrete random variable that takes on the values x n s 2 nrn 2 , n G 1, with probabilities pŽ n. s 1r2 n, n G 1. Then ⬁

1

ns1

n2

Ý

␮sEŽ X . s

,

which exists, because the series is convergent. However, ␮X2 does not exist, because ⬁ 2n ␮X2 s E Ž X 2 . s Ý 4 ns1 n and this series is divergent, since 2 nrn 4 ™ ⬁ as n ™ ⬁. Now, let Y be another discrete random variable that takes on the value zero with probability 12 and the values yn s 2 nq1 rn2 , n G 1, with probabilities q Ž n. s 1r2 nq1 , n G 1. Then, EŽ Y . s

⬁

1

ns1

n2

Ý

sEŽ X . .

The second noncentral moment of Y does not exist, since

␮X2 s E Ž Y 2 . s

⬁

2 nq1

ns1

n4

Ý

,

and this series is divergent. Since ␮X2 does not exist for both X and Y, none of their noncentral moments of order k ) 2 exist either, as can be seen from applying the comparison test. Thus X and Y have the same first noncentral moments, but do not have noncentral moments of any order greater than 1. These two random variables have obviously different distributions. 5.6.2. Moment and Probability Generating Functions Let X be a discrete random variable that takes on the values x 0 , x 1 , x 2 , . . . with probabilities pŽ n., n G 0. The Moment Generating Function of X This function is defined as

␾ Ž t . sEŽ et X . s

⬁

Ý e t x pŽ n. n

ns0

Ž 5.61 .

187

APPLICATIONS IN STATISTICS

provided that the series converges. In particular, if x n s n for n G 0, then

␾Ž t. s

⬁

Ý e t n p Ž n. ,

Ž 5.62 .

ns0

which is a power series in e t. If ␳ is the radius of convergence for this series, then by Theorem 5.4.4, ␾ Ž t . is a continuous function of t and has derivatives of all orders inside its interval of convergence. Since d k␾ Ž t . dt k

ts0

s E Ž X k . s ␮Xk ,

k s 1, 2, . . . ,

Ž 5.63 .

␾ Ž t ., when it exists, can be used to obtain all noncentral moments of X, which can completely determine the distribution of X by Theorem 5.6.1. From Ž5.63., by using Maclaurin’s expansion of ␾ Ž t ., we can obtain an expression for this function as a power series in t: ␾ Ž t . s ␾ Ž 0. q s1q

⬁

Ý

ns1

⬁

tn

ns1

n!

Ý

␮Xn n!

␾ Ž n. Ž 0 .

t n.

Let us now go back to the series in Ž5.62.. If

lim

n™⬁

p Ž n q 1. pŽ n.

s p,

then by Theorem 5.4.2, the radius of convergence ␳ is

°1rp, ¢⬁,

␳ s~0,

0 - p- ⬁, ps ⬁, ps 0.

Alternatively, if lim sup n™⬁w pŽ n.x1r n s q, then

°1rq, ¢⬁,

␳ s~0,

0 - q - ⬁, q s ⬁, q s 0.

Ž 5.64 .

188

INFINITE SEQUENCES AND SERIES

For example, for the Poisson distribution, where p Ž n. s

ey␭␭ n n!

n s 0, 1, 2, . . . ,

,

we have lim n™⬁w pŽ n q 1.rpŽ n.x s lim n™⬁w ␭rŽ n q 1.x s 0. Hence, ␳ s ⬁, that is, ⬁

Ý

␾Ž t. s

ey␭␭ n n!

ns0

etn

converges uniformly for any value of t for which e t - ⬁, that is, y⬁ - t - ⬁. As a matter of fact, a closed-form expression for ␾ Ž t . can be found, since

Ž ␭et .

⬁

Ý

␾ Ž t . s ey␭

n

n!

ns0

s ey␭ exp Ž ␭ e t . s exp Ž ␭ e t y ␭ .

Ž 5.65 .

for all t.

The kth noncentral moment of X is then given by

␮Xk s

d k␾ Ž t .

s

dt k

d k Ž ␭ e t y ␭. dt k

ts0

. ts0

In particular, the first two noncentral moments are

␮X1 s ␮ s ␭ , ␮X2 s ␭ q ␭2 . This confirms our earlier finding concerning these two moments. It should be noted that formula Ž5.63. is valid provided that there exists a ␦ ) 0 such that the neighborhood N␦ Ž0. is contained inside the interval of convergence. For example, let X have the probability mass function p Ž n. s

6

n s 1, 2, . . . .

,

␲ 2 n2

Then lim

p Ž n q 1. p Ž n.

n™⬁

s 1.

Hence, by Theorem 5.4.4, the series

␾Ž t. s

⬁

6

ns1

␲ 2 n2

Ý

etn

189

APPLICATIONS IN STATISTICS

converges uniformly for values of t satisfying e t F r, where r F 1, or equivalently, for t F log r F 0. If, however, t ) 0, then the series diverges. Thus there does not exist a neighborhood N␦ Ž0. that is contained inside the interval of convergence for any ␦ ) 0. Consequently, formula Ž5.63. does not hold in this case. From the moment generating function we can derive a series of constants that play a role similar to that of the moments. These constants are called cumulants. They have properties that are, in certain circumstances, more useful than those of the moments. Cumulants were originally defined and studied by Thiele Ž1903.. By definition, the cumulants of X, denoted by ␬ 1 , ␬ 2 , . . . , ␬ n , . . . are constants that satisfy the following identity in t:

ž

exp ␬ 1 t q

␬2 t 2 2!

q ⭈⭈⭈ q

s 1 q ␮X1 t q

␬n t n n!

q ⭈⭈⭈

␮X2 2

␮Xn

2!

n!

t q ⭈⭈⭈ q

/

t n q ⭈⭈⭈ .

Ž 5.66 .

Using formula Ž5.64., this identity can be written as ⬁

Ý

ns1

␬n n!

t n s log ␾ Ž t . ,

Ž 5.67 .

provided that ␾ Ž t . exists and is positive. By definition, the natural logarithm of the moment generating function of X is called the cumulant generating function. Formula Ž5.66. can be used to express the noncentral moments in terms of the cumulants, and vice versa. Kendall and Stuart Ž1977, Section 3.14. give a general relationship that can be used for this purpose. For example,

␬ 1 s ␮X1 , ␬ 2 s ␮X2 y ␮X12 , ␬ 3 s ␮X3 y 3 ␮X1 ␮X2 q 2 ␮X13 . The cumulants have an interesting property in that they are, except for ␬ 1 , invariant to any constant shift c in X. That is, for n s 2, 3, . . . , ␬ n is not changed if X is replaced by X q c. This follows from noting that E w e Ž Xqc .t x s e ct␾ Ž t . , which is the moment generating function of X q c. But log e ct␾ Ž t . s ct q log ␾ Ž t . .

190

INFINITE SEQUENCES AND SERIES

By comparison with Ž5.67. we can then conclude that except for ␬ 1 , the cumulants of X q c are the same as those of X. This contrasts sharply with the noncentral moments of X, which are not invariant to such a shift. Another advantage of using cumulants is that they can be employed to obtain approximate expressions for the percentile points of the distribution of X Žsee Section 9.5.1.. EXAMPLE 5.6.1. Let X be a Poisson random variable whose moment generating function is given by formula Ž5.65.. By applying Ž5.67. we get ⬁

Ý

ns1

␬n n!

t n s log exp Ž ␭ e t y ␭ . s ␭et y ␭ s␭

⬁

tn

ns1

n!

Ý

.

Here, we have made use of Maclaurin’s expansion of e t. This series converges for any value of t. It follows that ␬ n s ␭ for n s 1, 2, . . . . The Probability Generating Function This is similar to the moment generating function. It is defined as

␺ Ž t . sEŽ t X . s

⬁

Ý t x pŽ n. . n

ns0

In particular, if x n s n for n G 0, then

␺ Ž t. s

⬁

Ý t n p Ž n. ,

Ž 5.68 .

ns0

which is a power series in t. Within its interval of convergence, this series represents a continuous function with derivatives of all orders. We note that ␺ Ž0. s pŽ0. and that 1 d k␺ Ž t . k!

dt k

ts0

s pŽ k . ,

k s 1, 2, . . . .

Ž 5.69 .

Thus, the entire probability distribution of X is completely determined by ␺ Ž t .. The probability generating function is also useful in determining the

191

APPLICATIONS IN STATISTICS

moments of X. This is accomplished by using the relation d k␺ Ž t . dt k

s ts1

⬁

Ý n Ž n y 1. ⭈⭈⭈ Ž n y k q 1. p Ž n .

nsk

s E X Ž X y 1 . ⭈⭈⭈ Ž X y k q 1 . .

Ž 5.70 .

The quantity on the right-hand side of Ž5.70. is called the kth factorial moment of X, which we denote by ␪ k . The noncentral moments of X can be derived from the ␪ k ’s. For example,

␮X1 s ␪ 1 , ␮X2 s ␪ 2 q ␪ 1 , ␮X3 s ␪ 3 q 3␪ 2 q ␪ 1 , ␮X4 s ␪4 q 6␪ 3 q 7␪ 2 q ␪ 1 . Obviously, formula Ž5.70. is valid provided that t s 1 belongs to the interval of convergence of the series in Ž5.68.. If a closed-form expression is available for the moment generating function, then a corresponding expression can be obtained for ␺ Ž t . by replacing e t with t. For example, from formula Ž5.65., the probability generating function for the Poisson distribution is given by ␺ Ž t . s expŽ ␭ t y ␭.. 5.6.3. Some Limit Theorems In Section 3.7 we defined convergence in probability of a sequence of random variables. In Section 4.5.1 convergence in distribution of the same sequence was introduced. In this section we introduce yet another type of convergence. Definition 5.6.1. A sequence
X n4⬁ns1 of random variables converges in quadratic mean to a random variable X if lim E Ž X n y X . s 0. 2

n™⬁

q.m.

X.

6

This convergence is written symbolically as X n

I

6

Convergence in quadratic mean implies convergence in probability. This q.m. follows directly from applying Markov’s inequality: If X n X, then for any ⑀ ) 0, P Ž < Xn y X < G ⑀ . F

1

⑀2

E Ž Xn y X . ™ 0 2

as n ™ ⬁. This shows that the sequence
X n4⬁ns1 converges in probability to X.

192

INFINITE SEQUENCES AND SERIES

5.6.3.1. The Weak Law of Large Numbers (Khinchine’s Theorem) Let
X i 4⬁is1 be a sequence of independent and identically distributed random variables with a finite mean ␮. Then X n converges in probability to ␮ as n ™ ⬁, where X n s Ž1rn.Ý nis1 X i is the sample mean of a sample of size n. Proof. See, for example, Lindgren Ž1976, Section 2.5.1. or Rao Ž1973, Section 2c.3.. I

6

Definition 5.6.2. A sequence
X n4⬁ns1 of random variables converges strongly, or almost surely, to a random variable X, written symbolically as a.s. Xn X, if for any ⑀ ) 0,

ž

/

lim P sup < X n y X < ) ⑀ s 0.

N™⬁

nGN

I

Theorem 5.6.2. Let
X n4⬁ns1 be a sequence of random variables. Then we have the following: a.s.

c, where c is constant, then X n converges in probability to c. a.s. c, and the series Ý⬁ns1 E Ž X n y c . 2 converges, then X n c.

6 q.m.

6

6

1. If X n 2. If X n

5.6.3.2. The Strong Law of Large Numbers (Kolmogoro©’s Theorem)

6

Let
X n4⬁ns1 be a sequence of independent random variables such that E Ž X n . s ␮ n and VarŽ X n . s ␴n2 , n s 1, 2, . . . . If the series Ý⬁ns1 ␴n2rn2 cona.s. ␮ n , where ␮ n s Ž1rn.Ý nis1 ␮ i . verges, then X n Proof. See Rao Ž1973, Section 2c.3..

I

5.6.3.3. The Continuity Theorem for Probability Generating Functions See Feller Ž1968, page 280.. Suppose that for every k G 1, the sequence
pk Ž n.4⬁ns0 represents a discrete probability distribution. Let ␺ k Ž t . s Ý⬁ns0 t n pk Ž n. be the corresponding probability generating function Ž k s 1, 2, . . . .. In order for a limit qn s lim pk Ž n . k™⬁

to exist for every n s 0, 1, . . . , it is necessary and sufficient that the limit

␺ Ž t . s lim ␺ k Ž t . k™⬁

193

APPLICATIONS IN STATISTICS

exist for every t in the open interval Ž0, 1.. In this case,

␺ Ž t. s

⬁

Ý t n qn .

ns0

This theorem implies that a sequence of discrete probability distributions converges if and only if the corresponding probability generating functions converge. It is important here to point out that the qn’s may not form a discrete probability distribution Žbecause they may not sum to 1.. The function ␺ Ž t . may not therefore be a probability generating function. 5.6.4. Power Series and Logarithmic Series Distributions The power series distribution, which was introduced by Kosambi Ž1949., represents a family of discrete distributions, such as the binomial, Poisson, and negative binomial. Its probability mass function is given by an ␪ n

p Ž n. s

fŽ␪ .

n s 0, 1,2, . . . ,

,

where a n G 0, ␪ ) 0, and f Ž ␪ . is the function fŽ␪ . s

⬁

Ý an ␪ n .

Ž 5.71 .

ns0

This function is defined provided that ␪ falls inside the interval of convergence of the series in Ž5.71.. For example, for the Poisson distribution, ␪ s ␭, where ␭ is the mean, a n s 1rn! for n s 0, 1, 2, . . . , and f Ž ␪ . s e ␭. For the negative binomial, ␪ s 1 y p and a n s n q nr y 1 , n s 0, 1, 2, . . . , where n s number of failures, r s number of successes, and ps probability of success on each trial, and thus

ž

/

fŽ␪ . s

⬁

Ý

ns0

ž

/

1 n nqry1 Ž1 y p. s r . n p

A special case of the power series distribution is the logarithmic series distribution. It was first introduced by Fisher, Corbet, and Williams Ž1943. while studying abundance and diversity for insect trap data. The probability mass function for this distribution is p Ž n. s y where 0 - ␪ - 1.

␪n n log Ž 1 y ␪ .

,

n s 1, 2, . . . ,

194

INFINITE SEQUENCES AND SERIES

The logarithmic series distribution is useful in the analysis of various kinds of data. A description of some of its applications can be found, for example, in Johnson and Kotz Ž1969, Chapter 7.. 5.6.5. Poisson Approximation to Power Series Distributions Ž1991.. See Perez-Abreu ´ The Poisson distribution can provide an approximation to the distribution of the sum of random variables having power series distributions. This is based on the following theorem: Theorem 5.6.3. For each k G 1, let X 1 , X 2 , . . . , X k be independent nonnegative integer-valued random variables with a common power series distribution pk Ž n . s a n ␪ knrf Ž ␪ k . ,

n s 0, 1, . . . ,

where a n G 0 Ž n s 0, 1, . . . . are independent of k and f Ž ␪k . s

⬁

Ý an ␪ kn ,

␪ k ) 0.

ns0

Let a0 ) 0, ␭ ) 0 be fixed and Sk s Ý kis1 X i . If k ␪ k ™ ␭ as k ™ ⬁, then lim P Ž Sk s n . s ey␭ 0␭ 0nrn!,

n s 0, 1, . . . ,

k™⬁

where ␭0 s ␭ a1ra0 . Ž1991, page 43.. Proof. See Perez-Abreu ´

I

By using this theorem we can obtain the well-known Poisson approximation to the binomial and the negative binomial distributions as shown below. EXAMPLE 5.6.2 ŽThe Binomial Distribution .. For each k G 1, let X 1 , . . . , X k be a sequence of independent Bernoulli random variables with success probability pk . Let Sk s Ý kis1 X i . Suppose that kpk ™ ␭ ) 0 as k ™ ⬁. Then, for each n s 0, 1, . . . , lim P Ž Sk s n . s lim

k™⬁

k™⬁

ž/

k n kyn p Ž 1 y pk . n k

s ey␭␭ nrn!. This follows from the fact that the Bernoulli distribution with success probability pk is a power series distribution with ␪ k s pkrŽ1 y pk . and f Ž ␪ k . s

195

APPLICATIONS IN STATISTICS

1 q ␪ k . Since a0 s a1 s 1, and k ␪ k ™ ␭ as k ™ ⬁, we get from applying Theorem 5.6.3 that lim P Ž S k s n . s ey␭␭ nrn!.

k™⬁

EXAMPLE 5.6.3 ŽThe Negative Binomial Distribution .. We recall that a random variable Y has the negative binomial distribution if it represents the number of failures n Žin repeated trials. before the kth success Ž k G 1.. Let pk denote the probability of success on a single trial. Let X 1 , X 2 , . . . , X k be random variables defined as X 1 s number of failures occurring before the 1st success, X 2 s number of failures occurring between the 1st success and the 2nd success, . . . X k s number of failures occurring between the Ž k y 1.st success and the kth success. Such random variables have what is known as the geometric distribution. It is a special case of the negative binomial distribution with k s 1. The common probability distribution of the X i ’s is n

P Ž X i s n . s pk Ž 1 y pk . ,

n s 0, 1 . . . ;

i s 1, 2, . . . , k.

This is a power series distribution with a n s 1 Ž n s 0, 1, . . . ., ␪ k s 1 y pk , and f Ž ␪k . s

⬁

1

1

n Ý Ž 1 y pk . s 1 y Ž 1 y p . s p . k k ns0

It is easy to see that X 1 , X 2 , . . . , X k are independent and that Y s Sk s Ý kis1 X i . Let us now assume that k Ž1 y pk . ™ ␭ ) 0 as k ™ ⬁. Then from Theorem 5.6.3 we obtain the following result: lim P Ž Sk s n . s ey␭␭ nrn!,

k™⬁

5.6.6. A Ridge Regression Application Consider the linear model y s X␤ q ⑀ ,

n s 0, 1. . . . .

196

INFINITE SEQUENCES AND SERIES

where y is a vector of n response values, X is an n = p matrix of rank p, ␤ is a vector of p unknown parameters, and ⑀ is a random error vector such that E Ž ⑀ . s 0 and VarŽ ⑀ . s ␴ 2 I n . All variables in this model are corrected for their means and scaled to unit length, so that X⬘X and X⬘y are in correlation form. We recall from Section 2.4.2 that if the columns of X are multicollinear, ˆ s ŽX⬘X.y1 X⬘y, is an unrelithen the least-squares estimator of ␤, namely ␤ able estimator due to large variances associated with its elements. There are several methods that can be used to combat multicollinearity. A review of such methods can be found in Ofir and Khuri Ž1986.. Ridge regression is one of the most popular of these methods. This method, which was developed by Hoerl and Kennard Ž1970a, b., is based on adding a positive constant k to the diagonal elements of X⬘X. This leads to a biased estimator ␤* of ␤ called the ridge regression estimator and is given by ␤* s Ž X⬘X q kI n .

y1

X⬘y.

The elements of ␤* can have substantially smaller variances than the correˆ Žsee, for example, Montgomery and Peck, 1982, sponding elements of ␤ . Section 8.5.3 . Draper and Herzberg Ž1987. showed that the ridge regression residual sum of squares can be represented as a power series in k. More specifically, consider the vector of predicted responses,

ˆyk s X␤* s X Ž X⬘X q kI n .

y1

Ž 5.72 .

X⬘y,

which is based on using ␤*. Formula Ž5.72. can be written as

ˆyk s X Ž X⬘X .

y1

I n q k Ž X⬘X .

y1 y1

X⬘y.

Ž 5.73 .

From Theorem 5.5.2, if all the eigenvalues of k ŽX⬘X.y1 are less than one in absolute value, then I n q k Ž X⬘X .

y1 y1

s

⬁

Ý Ž y1. i k i Ž X⬘X. yi .

Ž 5.74 .

is0

From Ž5.73. and Ž5.74. we get

ˆyk s Ž H 1 y kH 2 q k 2 H 3 y k 3 H 4 q ⭈⭈⭈ . y, where H i s XŽX⬘X.yi X⬘, i G 1. Thus the ridge regression residual sum of squares, which is the sum of squares of deviations of the elements of y from the corresponding elements of ˆ yk , is

Ž y yˆyk . ⬘ Ž y yˆyk . s y⬘Qy,

197

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

where Q s Ž I n y H 1 q kH 2 y k 2 H 3 q k 3 H 4 y ⭈⭈⭈ . . 2

It can be shown Žsee Exercise 5.32. that y⬘Qys SS E q

⬁

Ý Ž i y 2. Ž yk . iy1 Si ,

Ž 5.75 .

is3

where SS E is the usual least-squares residual sum of squares, which can be obtained when k s 0, that is, SS E s y⬘ I n y X Ž X⬘X .

y1

X⬘ y,

and Si s y⬘H i y, i G 3. The terms to the right of Ž5.75., other than SS E , are bias sums of squares induced by the presence of a nonzero k. Draper and Herzberg Ž1987. demonstrated by means of an example that the series in Ž5.75. may diverge or else converge very slowly, depending on the value of k.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Apostol, T. M. Ž1964.. Mathematical Analysis. Addison-Wesley, Reading, Massachusetts. ŽInfinite series are discussed in Chap. 12.. Boyer, C. B. Ž1968.. A History of Mathematics. Wiley, New York. Draper, N. R., and A. M. Herzberg Ž1987.. ‘‘A ridge-regression sidelight.’’ Amer. Statist., 41, 282᎐283. Draper, N. R., and H. Smith Ž1981.. Applied Regression Analysis, 2nd ed. Wiley, New York. ŽChap. 6 discusses ridge regression in addition to the various statistical procedures for selecting variables in a regression model.. Feller, W. Ž1968.. An Introduction to Probability Theory and Its Applications, Vol. I, 3rd ed. Wiley, New York. Fisher, R. A., and E. A. Cornish Ž1960.. ‘‘The percentile points of distribution having known cumulants.’’ Technometrics, 2, 209᎐225. Fisher, R. A., A. S. Corbet, and C. B. Williams Ž1943.. ‘‘The relation between the number of species and the number of individuals in a random sample of an animal population.’’ J. Anim. Ecology, 12, 42᎐58. Fisz, M. Ž1963.. Probability Theory and Mathematical Statistics, 3rd ed. Wiley, New York. ŽChap. 5 deals almost exclusively with limit distributions for sums of independent random variables. . Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. ŽChap. 2 discusses limits of sequences; Chap. 13 deals with infinite series of constant terms; Chap. 14 deals with sequences and series of functions; Chap. 15 provides a study of power series. . Gantmacher, F. R. Ž1959.. The Theory of Matrices, Vol. I. Chelsea, New York.

198

INFINITE SEQUENCES AND SERIES

Graybill, F. A. Ž1983.. Matrices with Applications in Statistics, 2nd ed. Wadsworth, Belmont, California. ŽChap. 5 includes a section on sequences and series of matrices. . Hirschman, I. I., Jr. Ž1962.. Infinite Series. Holt, Rinehart and Winston, New York. ŽThis book is designed to be used in applied courses beyond the advanced calculus level. It emphasizes applications of the theory of infinite series. . Hoerl, A. E., and R. W. Kennard Ž1970a.. ‘‘Ridge regression: Biased estimation for non-orthogonal problems.’’ Technometrics, 12, 55᎐67. Hoerl, A. E., and R. W. Kennard Ž1970b.. ‘‘Ridge regression: Applications to non-orthogonal problems.’’ Technometrics, 12, 69᎐82; Correction. 12, 723. Hogg, R. V., and A. T. Craig Ž1965.. Introduction to Mathematical Statistics, 2nd ed. Macmillan, New York. Hyslop, J. M. Ž1954.. Infinite Series, 5th ed. Oliver and Boyd, Edinburgh. ŽThis book presents a concise treatment of the theory of infinite series. It provides the basic elements of this theory in a clear and easy-to-follow manner.. Johnson, N. L., and S. Kotz Ž1969.. Discrete Distributions. Houghton Mifflin, Boston. ŽChaps. 1 and 2 contain discussions concerning moments, cumulants, generating functions, and power series distributions. . Kendall, M., and A. Stuart Ž1977.. The Ad®anced Theory of Statistics, Vol. 1, 4th ed. Macmillan, New York. ŽMoments, cumulants, and moment generating functions are discussed in Chap. 3.. Knopp, K. Ž1951.. Theory and Application of Infinite Series. Blackie and Son, London. ŽThis reference book provides a detailed and comprehensive study of the theory of infinite series. It contains many interesting examples.. Kosambi, D. D. Ž1949.. ‘‘Characteristic properties of series distributions.’’ Proc. Nat. Inst. Sci. India, 15, 109᎐113. Lancaster, P. Ž1969.. Theory of Matrices. Academic Press, New York. ŽChap. 5 discusses functions of matrices in addition to sequences and series involving matrix terms.. Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽChap. 2 contains a section on moments of a distribution and a proof of Markov’s inequality. . Montgomery, D. C., and E. A. Peck Ž1982.. Introduction to Linear Regression Analysis. Wiley, New York. ŽChap. 8 discusses the effect of multicollinearity and the methods for dealing with it including ridge regression.. Nurcombe, J. R. Ž1979.. ‘‘A sequence of convergence tests.’’ Amer. Math. Monthly, 86, 679᎐681. Ofir, C., and A. I. Khuri Ž1986.. ‘‘Multicollinearity in marketing models: Diagnostics and remedial measures.’’ Internat. J. Res. Market., 3, 181᎐205. ŽThis is a review article that surveys the problem of multicollinearity in linear models and the various remedial measures for dealing with it.. Perez-Abreu, V. Ž1991.. ‘‘Poisson approximation to power series distributions.’’ Amer. ´ Statist., 45, 42᎐45. Pye, W. C., and P. G. Webster Ž1989.. ‘‘A note on Raabe’s test extended.’’ Math. Comput. Ed., 23, 125᎐128.

199

EXERCISES

Rao, C. R. Ž1973.. Linear Statistical Inference and Its Applications, 2nd ed. Wiley, New York. ŽChap. 2 contains a section on limit theorems in statistics, including the weak and strong laws of large numbers.. Rudin, W. Ž1964.. Principles of Mathematical Analysis, 2nd ed. McGraw-Hill, New York. ŽSequences and series of scalar constants are discussed in Chap. 3; sequences and series of functions are studied in Chap. 7.. Thiele, T. N. Ž1903.. Theory of Obser®ations. Layton, London. Reprinted in Ann. Math. Statist., 2, 165᎐307 Ž1931.. Wilks, S. S. Ž1962.. Mathematical Statistics. Wiley, New York. ŽChap. 4 discusses different types of convergence of random variables; Chap. 5 presents several results concerning the moments of a distribution. . Withers, C. S. Ž1984.. ‘‘Asymptotic expansions for distributions and quantiles with power series cumulants.’’ J. Roy. Statist. Soc. Ser. B, 46, 389᎐396.

EXERCISES In Mathematics 5.1. Suppose that
a n4⬁ns1 is a bounded sequence of positive terms. (a) Define bn s max
a1 , a2 , . . . , a n4 , n s 1, 2, . . . . Show that the sequence
bn4⬁ns1 converges, and identify its limit. (b) Suppose further that a n ™ c as n ™ ⬁, where c ) 0. Show that c n ™ c, where
c n4⬁ns1 is the sequence of geometric means, c n s n ŽŁ is1 a i .1r n. 5.2. Suppose that
a n4⬁ns1 and
bn4⬁ns1 are any two Cauchy sequences. Let d n s < a n y bn < , n s 1, 2, . . . . Show that the sequence
d n4⬁ns1 converges. 5.3. Prove Theorem 5.1.3. 5.4. Show that if
a n4⬁ns1 is a bounded sequence, then the set E of all its subsequential limits is also bounded. 5.5. Suppose that a n ™ c as n ™ ⬁ and that
a i 4⬁is1 is a sequence of positive terms for which Ý nis1 ␣ i ™ ⬁ as n ™ ⬁. (a) Show that Ý nis1 ␣ i a i Ý nis1 ␣ i

™c

as n ™ ⬁.

In particular, if ␣ i s 1 for all i, then 1 n

n

Ý ai ™ c

as n ™ ⬁.

is1

(b) Show that the converse of the special case in Ža. does not always

200

INFINITE SEQUENCES AND SERIES

hold by giving a counterexample of a sequence
a n4⬁ns1 that does not converge, yet ŽÝ nis1 a i .rn converges as n ™ ⬁. 5.6. Let
a n4⬁ns1 be a sequence of positive terms such that a nq1 an

™b

as n ™ ⬁,

where 0 - b- 1. Show that there exist constants c and r such that 0 - r - 1 and c ) 0 for which a n - cr n for sufficiently large values of n. 5.7. Suppose that we have the sequence
a n4⬁ns1 , where a1 s 1 and a nq1 s

a n Ž 3bq a2n . 3a2n q b

b) 0,

,

n s 1, 2, . . . .

Show that the sequence converges, and find its limit. 5.8. Show that the sequence
a n4⬁ns1 converges, and find its limit, where a1 s 1 and a nq1 s Ž 2 q a n .

1r2

,

n s 1, 2, . . . .

5.9. Let
a n4⬁ns1 be a sequence and sn s Ý nis1 a i . (a) Show that lim sup n™⬁Ž snrn. F lim sup n™⬁ a n . (b) If snrn converges as n ™ ⬁, then show that a nrn™ 0 as n ™ ⬁. 5.10. Show that the sequence
a n4⬁ns1 , where a n s Ý nis1 Ž1ri ., is not a Cauchy sequence and is therefore divergent. 5.11. Suppose that the sequence
a n4⬁ns1 satisfies the following condition: There is an r, 0 - r - 1, such that < a nq1 y a n < - br n ,

n s 1, 2, . . . ,

where b is a positive constant. Show that this sequence converges. 5.12. Show that if a n G 0 for all n, then Ý⬁ns1 a n converges if and only if
sn4⬁ns1 is a bounded sequence, where sn is the nth partial sum of the series. 5.13. Show that the series Ý⬁ns1 w1rŽ3n y 1.Ž3n q 2.x converges to 16 . 5.14. Show that the series Ý⬁ns1 Ž n1r n y 1. p is divergent for pF 1. 5.15. Let Ý⬁ns1 a n be a divergent series of positive terms. (a) If
a n4⬁ns1 is a bounded sequence, then show that Ý⬁ns1 w a nrŽ1 q a n .x diverges.

201

EXERCISES

(b) Show that Ža. is true even if
a n4⬁ns1 is not a bounded sequence. 5.16. Let Ý⬁ns1 a n be a divergent series of positive terms. Show that an sn2

F

1 sny1

1

y

sn

n s 2, 3, . . . ,

,

where sn is the nth partial sum of the series; then deduce that Ý⬁ns1 Ž a nrsn2 . converges. 5.17. Let Ý⬁ns1 a n be a convergent series of positive terms. Let rn s Ý⬁isn a i . Show that for m - n, n

ai

Ý

ri

ism

)1y

rn rm

,

and deduce that Ý⬁ns1 Ž a nrrn . diverges. 5.18. Given the two series Ý⬁ns1 Ž1rn., Ý⬁ns1 Ž1rn2 .. Show that lim

ž / ž / 1

1

n2

n™⬁

s 1,

n

n™⬁

lim

1rn

1rn

s 1.

5.19. Test for convergence of the series Ý⬁ns1 a n , where n

Ž a.

a n s Ž n1r n y 1 . ,

Ž b.

an s

Ž c.

an s

Ž d.

a n s n q 'n y 'n ,

log Ž 1 q n . log Ž 1 q e n

2

.

,

1 = 3 = 5 = ⭈⭈⭈ = Ž 2 n y 1 .

'

2 = 4 = 6 = ⭈⭈⭈ = 2 n

n

Ž y1. 4 n

Ž e.

an s

Ž f.

a n s sin

nn

ž

nq

, 1 n

/

␲ .

⭈

1 2nq1

,

202

INFINITE SEQUENCES AND SERIES

5.20. Determine the values of x for which each of the following series converges uniformly: ⬁

nq2

ns1

3n

Ý

Ž a.

⬁

10 n

ns1

n

Ý

Ž b.

x2n,

x n,

⬁

Ý Ž n q 1. 2 x n ,

Ž c.

ns1

cos Ž nx .

⬁

. Ý 2 ns1 n Ž n q 1 .

Ž d.

5.21. Consider the series ⬁

⬁

ns1

ns1

Ý an s Ý

Ž y1. n

ny1

.

Let Ý⬁ns1 bn be a certain rearrangement of Ý⬁ns1 a n given by 1 q 13 y 21 q 51 q 71 y 41 q 91 q 111 y 61 q ⭈⭈⭈ , where two positive terms are followed by one negative. Show that the sum of the original series is less than 10 12 , whereas that of the rearranged series Žwhich is convergent. exceeds 11 12 . 5.22. Consider Cauchy’s product of Ý⬁ns0 a n with itself, where an s

Ž y1.

n

'n q 1

n s 0, 1, 2, . . . .

,

Show that this product is divergent. w Hint: Show that the nth term of this product does not go to zero as n ™ ⬁.x 5.23. Consider the sequence of functions
f nŽ x .4⬁ns1 , where for n s 1, 2, . . . fn Ž x . s

nx 1 q nx 2

,

xG 0.

Find the limit of this sequence, and determine whether or not the convergence is uniform on w0, ⬁..

203

EXERCISES

5.24. Consider the series Ý⬁ns1 Ž1rn x .. (a) Show that this series converges uniformly on w1 q ␦ , ⬁., where ␦ is any positive number. w Note: The function represented by this series is known as Riemann’s ␨-function and is denoted by ␨ Ž x ..x (b) Is ␨ Ž x . differentiable on w1 q ␦ , ⬁.? If so, give a series expansion for ␨ ⬘Ž x .. In Statistics 5.25. Prove formulas Ž5.56. and Ž5.57.. 5.26. Find a series expansion for the moment generating function of the negative binomial distribution. For what values of t does this series converge uniformly? In this case, can formula Ž5.63. be applied to obtain an expression for the kth noncentral moment Ž k s 1, 2, . . . . of this distribution? Why or why not? 5.27. Find the first three cumulants of the negative binomial distribution. 5.28. Show that the moments ␮Xn Ž n s 1, 2, . . . . of a random variable X determine the cumulative distribution functions of X uniquely if

lim sup

< ␮Xn < 1r n n

n™⬁

is finite.

w Hint: Use the fact that n!; '2␲ n nq1r2 eyn as n ™ ⬁.x 5.29. Find the moment generating function of the logarithmic series distribution, and deduce that the mean and variance of this distribution are given by

␮ s ␣␪r Ž 1 y ␪ . , ␴ 2s␮

ž

1 1y␪

/

y␮ ,

where ␣ s y1rlogŽ1 y ␪ .. 5.30. Let
X n4⬁ns1 be a sequence of binomial random variables where the probability mass function of X n Ž n s 1, 2, . . . . is given by pn Ž k . s

ž/

n k nyk p Ž1 y p. , k

k s 0, 1,2, . . . , n,

204

INFINITE SEQUENCES AND SERIES

where 0 - p- 1. Further, let the random variable Yn be defined as Yn s

Xn n

y p.

(a) Show that E Ž X n . s np and VarŽ X n . s npŽ1 y p .. (b) Apply Chebyshev’s inequality to show that P Ž < Yn < G ⑀ . F

pŽ1 y p. n⑀ 2

,

where ⑀ ) 0. (c) Deduce from Žb. that Yn converges in probability to zero. w Note: This result is known as Bernoulli’s law of large numbers.x 5.31. Let X 1 , X 2 , . . . , X n be a sequence of independent Bernoulli random variables with success probability pn . Let Sn s Ý nis1 X i . Suppose that npn ™ ␮ ) 0 as n ™ ⬁. (a) Give an expression for ␾nŽ t ., the moment generating function of Sn . (b) Show that lim ␾n Ž t . s exp Ž ␮ e t y ␮ . ,

n™⬁

which is the moment generating function of a Poisson distribution with mean ␮. 5.32. Prove formula Ž5.75..

CHAPTER 6

Integration

The origin of integral calculus can be traced back to the ancient Greeks. They were motivated by the need to measure the length of a curve, the area of a surface, or the volume of a solid. Archimedes used techniques very similar to actual integration to determine the length of a segment of a curve. Democritus Ž410 B.C.. had the insight to consider that a cone was made up of infinitely many plane cross sections parallel to the base. The theory of integration received very little stimulus after Archimedes’s remarkable achievements. It was not until the beginning of the seventeenth century that the interest in Archimedes’s ideas began to develop. Johann Kepler Ž1571᎐1630. was the first among European mathematicians to develop the ideas of infinitesimals in connection with integration. The use of the term ‘‘integral’’ is due to the Swiss mathematician Johann Bernoulli Ž1667᎐1748.. In the present chapter we shall study integration of real-valued functions of a single variable x according to the concepts put forth by the German mathematician Georg Friedrich Riemann Ž1826᎐1866.. He was the first to establish a rigorous analytical foundation for integration, based on the older geometric approach. 6.1. SOME BASIC DEFINITIONS Let f Ž x . be a function defined and bounded on a finite interval w a, b x. Suppose that this interval is partitioned into a finite number of subintervals by a set of points Ps
x 0 , x 1 , . . . , x n4 such that as x 0 - x 1 - x 2 - ⭈⭈⭈ - x n s b. This set is called a partition of w a, b x. Let ⌬ x i s x i y x iy1 Ž i s 1, 2, . . . , n., and ⌬ p be the largest of ⌬ x 1 , ⌬ x 2 , . . . , ⌬ x n . This value is called the norm of P. Consider the sum SŽ P , f . s

n

Ý f Ž ti . ⌬ x i ,

is1

where t i is a point in the subinterval w x iy1 , x i x, i s 1, 2, . . . , n. 205

206

INTEGRATION

The function f Ž x . is said to be Riemann integrable on w a, b x if a number A exists with the following property: For any given ⑀ ) 0 there exists a number ␦ ) 0 such that AySŽ P , f . -⑀ for any partition P of w a, b x with a norm ⌬ p - ␦ , and for any choice of the point t i in w x iy1 , x i x, i s 1, 2, . . . , n. The number A is called the Riemann integral of f Ž x . on w a, b x and is denoted by Hab f Ž x . dx. The integration symbol H was first used by the German mathematician Gottfried Wilhelm Leibniz Ž1646᎐1716. to represent a sum Žit was derived from the first letter of the Latin word summa, which means a sum..

6.2. THE EXISTENCE OF THE RIEMANN INTEGRAL In order to investigate the existence of the Riemann integral, we shall need the following theorem: Theorem 6.2.1. Let f Ž x . be a bounded function on a finite interval, w a, b x. For every partition Ps
x 0 , x 1 , . . . , x n4 of w a, b x, let m i and Mi be, respectively, the infimum and supremum of f Ž x . on w x iy1 , x i x, i s 1, 2, . . . , n. If, for a given ⑀ ) 0, there exists a ␦ ) 0 such that US P Ž f . y LSP Ž f . - ⑀

Ž 6.1 .

whenever ⌬ p - ␦ , where ⌬ p is the norm of P, and LSP Ž f . s

n

Ý mi ⌬ x i ,

is1

US P Ž f . s

n

Ý Mi ⌬ x i ,

is1

then f Ž x . is Riemann integrable on w a, b x. Conversely, if f Ž x . is Riemann integrable, then inequality Ž6.1. holds for any partition P such that ⌬ p - ␦ . wThe sums, LSP Ž f . and US P Ž f ., are called the lower sum and upper sum, respectively, of f Ž x . with respect to the partition P.x In order to prove Theorem 6.2.1 we need the following lemmas: Lemma 6.2.1. Let P and P⬘ be two partitions of w a, b x such that P⬘ > P Ž P⬘ is called a refinement of P and is constructed by adding partition points between those that belong to P .. Then US P ⬘ Ž f . F US P Ž f . , LSP ⬘ Ž f . G LSP Ž f . .

207

THE EXISTENCE OF THE RIEMANN INTEGRAL

Proof. Let Ps
x 0 , x 1 , . . . , x n4 . By the nature of the partition P⬘, the ith Ž2. Ž k i. subinterval ⌬ x i s x i y x iy1 is divided into k i parts ⌬Ž1. x i , ⌬ x i , . . . , ⌬ x i , where Ž j. Ž j. k i G 1, i s 1, 2, . . . , n. If m i and Mi denote, respectively, the infimum and supremum of f Ž x . on ⌬Žxj.i , then m i F mŽi j. F MiŽ j. F Mi for j s 1, 2, . . . , k i ; i s 1, 2, . . . , n, where m i and Mi are the infimum and supremum of f Ž x . on w x iy1 , x i x, respectively. It follows that LSP Ž f . s

n

Ý

is1

US P ⬘ Ž f . s

n

ki

n

mi ⌬ xi F

Ý Ý mŽi j.⌬Žxj. s LSP ⬘ Ž f . i

is1 js1 ki

n

Ý Ý MiŽ j.⌬Žxj. F Ý Mi ⌬ x i s USP Ž f . . i

is1 js1

I

is1

Lemma 6.2.2. Let P and P⬘ be any two partitions of w a, b x. Then LSP Ž f . F US P ⬘Ž f .. Proof. Let P n s Pj P⬘. The partition P⬙ is a refinement of both P and P⬘. Then, by Lemma 6.2.1, LSP Ž f . F LSP ⬙ Ž f . F US P ⬙ Ž f . F US P ⬘ Ž f . .

I

Proof of Theorem 6.2.1 Let ⑀ ) 0 be given. Suppose that inequality Ž6.1. holds for any partition P whose norm ⌬ p is less than ␦ . Let SŽ P, f . s Ý nis1 f Ž t i . ⌬ x i , where t i is a point in w x iy1 , x i x, i s 1, 2, . . . , n. By the definition of LSP Ž f . and US P Ž f . we can write LSP Ž f . F S Ž P , f . F US P Ž f . .

Ž 6.2 .

Let m and M be the infimum and supremum, respectively, of f Ž x . on w a, b x; then m Ž b y a . F LSP Ž f . F US P Ž f . F M Ž by a . .

Ž 6.3 .

Let us consider two sets of lower and upper sums of f Ž x . with respect to partitions P, P⬘, P⬙, . . . such that P; P⬘ ; P⬙ ; ⭈⭈⭈ . Then, by Lemma 6.2.1, the set of upper sums is decreasing, and the set of lower sums is increasing. Furthermore, because of Ž6.3., the set of upper sums is bounded from below by mŽ by a., and the set of lower sums is bounded from above by M Ž by a.. Hence, the infimum of US P Ž f . and the supremum of LSP Ž f . with respect to P do exist Žsee Theorem 1.5.1.. From Lemma 6.2.2 it is easy to deduce that sup LSP Ž f . F inf US P Ž f . . P

P

208

INTEGRATION

Now, suppose that for the given ⑀ ) 0 there exists a ␦ ) 0 such that US P Ž f . y LSP Ž f . - ⑀

Ž 6.4 .

for any partition whose norm ⌬ p is less than ␦ . We have that LSP Ž f . F sup LSP Ž f . F inf US P Ž f . F US P Ž f . . P

P

Ž 6.5 .

Hence, inf US P Ž f . y sup LSP Ž f . - ⑀ . P

P

Since ⑀ ) 0 is arbitrary, we conclude that if Ž6.1. is satisfied, then inf US P Ž f . s sup LSP Ž f . . P

Ž 6.6 .

P

Furthermore, from Ž6.2., Ž6.4., and Ž6.5. we obtain SŽ P , f . yA -⑀ , where A is the common value of inf P US P Ž f . and sup P LSP Ž f .. This proves that A is the Riemann integral of f Ž x . on w a, b x. Let us now show that the converse of the theorem is true, that is, if f Ž x . is Riemann integrable on w a, b x, then inequality Ž6.1. holds. If f Ž x . is Riemann integrable, then for a given ⑀ ) 0 there exists a ␦ ) 0 such that n

⑀

Ý f Ž ti . ⌬ x i y A - 3

Ž 6.7 .

is1

and n

⑀

Ý f Ž tXi . ⌬ x i y A - 3

Ž 6.8 .

is1

for any partition Ps
x 0 , x 1 , . . . , x n4 of w a, b x with a norm ⌬ p - ␦ , and any choices of t i , tXi in w x iy1 , x i x, i s 1, 2, . . . , n, where A s Hab f Ž x . dx. From Ž6.7. and Ž6.8. we then obtain n

Ý

f Ž t i . y f Ž tXi . ⌬ x i -

is1

2⑀ 3

.

Now, Mi y m i is the supremum of f Ž x . y f Ž x⬘. for x, x⬘ in w x iy1 , x i x, i s 1, 2, . . . , n. It follows that for a given ␩ ) 0 we can choose t i , tXi in w x iy1 , x i x so that f Ž t i . y f Ž tXi . ) Mi y m i y ␩ ,

i s 1, 2, . . . , n,

209

THE EXISTENCE OF THE RIEMANN INTEGRAL

for otherwise Mi y m i y ␩ would be an upper bound for f Ž x . y f Ž x⬘. for all x, x⬘ in w x iy1 , x i x, which is a contradiction. In particular, if ␩ s ⑀rw3Ž by a.x, then we can find t i , tXi in w x iy1 , x i x such that n

Ý Ž Mi y m i . ⌬ x i

US P Ž f . y LSP Ž f . s

is1 n

-

Ý

f Ž t i . y f Ž tXi . ⌬ x i q ␩ Ž by a .

is1

-⑀ . This proves the validity of inequality Ž6.1..

I

Corollary 6.2.1. Let f Ž x . be a bounded function on w a, b x. Then f Ž x . is Riemann integrable on w a, b x if and only if inf P US P Ž f . s sup p LSP Ž f ., where LSP Ž f . and US P Ž f . are, respectively, the lower and upper sums of f Ž x . with respect to a partition P of w a, b x. I

Proof. See Exercise 6.1.

EXAMPLE 6.2.1. Let f Ž x .: w0, 1x ™ R be the function f Ž x . s x 2 . Then, f Ž x . is Riemann integrable on w0, 1x. To show this, let Ps
x 0 , x 1 , . . . , x n4 be any partition of w0, 1x, where x 0 s 0, x n s 1. Then LSP Ž f . s

n

2 ⌬ xi , Ý x iy1

is1

US P Ž f . s

n

Ý x i2 ⌬ x i .

is1

Hence, US P Ž f . y LSP Ž f . s

n

2 . ⌬ xi Ý Ž x i2 y x iy1

is1

n

F⌬p

2 ., Ý Ž x i2 y x iy1

is1

where ⌬ p is the norm of P. But n

2 . s x n2 y x 02 s 1. Ý Ž x i2 y x iy1

is1

Thus US P Ž f . y LSP Ž f . F ⌬ p .

210

INTEGRATION

It follows that for a given ⑀ ) 0 we can choose ␦ s ⑀ such that for any partition P whose norm ⌬ p is less than ␦ , US P Ž f . y LSP Ž f . - ⑀ . By Theorem 6.2.1, f Ž x . s x 2 is Riemann integrable on w0, 1x. EXAMPLE 6.2.2. Consider the function f Ž x .: w0, 1x ™ R such that f Ž x . s 0 if x a rational number and f Ž x . s 1 if x is irrational. Since every subinterval of w0, 1x contains both rational and irrational numbers, then for any partition Ps
x 0 , x 1 , . . . , x n4 of w0, 1x we have US P Ž f . s LSP Ž f . s

n

n

is1

is1

n

n

is1

is1

Ý Mi ⌬ x i s Ý ⌬ x i s 1, Ý m i ⌬ x i s Ý 0 ⌬ x i s 0.

It follows that inf US P Ž f . s 1 P

and

sup LSP Ž f . s 0. P

By Corollary 6.2.1, f Ž x . is not Riemann integrable on w0, 1x.

6.3. SOME CLASSES OF FUNCTIONS THAT ARE RIEMANN INTEGRABLE There are certain classes of functions that are Riemann integrable. Identifying a given function as a member of such a class can facilitate the determination of its Riemann integrability. Some of these classes of functions include: Ži. continuous functions; Žii. monotone functions; Žiii. functions of bounded variation. Theorem 6.3.1. integrable there.

If f Ž x . is continuous on w a, b x, then it is Riemann

Proof. Since f Ž x . is continuous on a closed and bounded interval, then by Theorem 3.4.6 it must be uniformly continuous on w a, b x. Consequently, for a given ⑀ ) 0 there exists a ␦ ) 0 that depends only on ⑀ such that for any x 1 , x 2 in w a, b x we have f Ž x1 . y f Ž x 2 . -

⑀ by a

SOME CLASSES OF FUNCTIONS THAT ARE RIEMANN INTEGRABLE

211

if < x 1 y x 2 < - ␦ . Let Ps
x 0 , x 1 , . . . , x n4 be a partition of P with a norm ⌬ p - ␦ . Then US P Ž f . y LSP Ž f . s

n

Ý Ž Mi y m i . ⌬ x i ,

is1

where m i and Mi are, respectively, the infimum and supremum of f Ž x . on w x iy1 , x i x, i s 1, 2, . . . , n. By Corollary 3.4.1 there exist points ␰ i , ␩i in w x iy1 , x i x such that m i s f Ž ␰ i ., Mi s f Ž␩i ., i s 1, 2, . . . , n. Since < ␩i y ␰ i < F ⌬ p - ␦ for i s 1, 2, . . . , n, then US P Ž f . y LSP Ž f . s

n

Ý

f Ž ␩i . y f Ž ␰ i . ⌬ x i

is1

-

⑀ by a

n

Ý ⌬ xi s ⑀ .

is1

By Theorem 6.2.1 we conclude that f Ž x . is Riemann integrable on w a, b x. I It should be noted that continuity is a sufficient condition for Riemann integrability, but is not a necessary one. A function f Ž x . can have discontinuities in w a, b x and still remains Riemann integrable on w a, b x. For example, consider the function f Ž x. s

½

y1, 1,

y1 F x - 0, 0 F xF 1.

This function is discontinuous at xs 0. However, it is Riemann integrable on wy1, 1x. To show this, let ⑀ ) 0 be given, and let Ps
x 0 , x 1 , . . . , x n4 be a partition of wy1, 1x such that ⌬ p - ⑀r2. By the nature of this function, f Ž x i . y f Ž x iy1 . G 0, and the infimum and supremum of f Ž x . on w x iy1 , x i x are equal to f Ž x iy1 . and f Ž x i ., respectively, i s 1, 2, . . . , n. Hence, US P Ž f . y LSP Ž f . s

n

Ý Ž Mi y m i . ⌬ x i

is1 n

s

Ý

f Ž x i . y f Ž x iy1 . ⌬ x i

is1 n

-⌬p

Ý

f Ž x i . y f Ž x iy1 . s ⌬ p f Ž 1 . y f Ž y1 .

is1

-

⑀ 2

f Ž 1 . y f Ž y1 . s ⑀ .

212

INTEGRATION

The function f Ž x . is therefore Riemann integrable on wy1, 1x by Theorem 6.2.1. On the basis of this example it is now easy to prove the following theorem: Theorem 6.3.2. If f Ž x . is monotone increasing Žor monotone decreasing . on w a, b x, then it is Riemann integrable there. Theorem 6.3.2 can be used to construct a function that has a countable number of discontinuities in w a, b x and is also Riemann integrable Žsee Exercise 6.2.. 6.3.1. Functions of Bounded Variation Let f Ž x . be defined on w a, b x. This function is said to be of bounded variation on w a, b x if there exists a number M) 0 such that for any partition Ps
x 0 , x 1 , . . . , x n4 of w a, b x we have n

Ý < ⌬ f i < F M,

is1

where ⌬ f i s f Ž x i . y f Ž x iy1 ., i s 1, 2, . . . , n. Any function that is monotone increasing Žor decreasing . on w a, b x is also of bounded variation there. To show this, let f Ž x . be monotone increasing on w a, b x. Then n

n

is1

is1

Ý < ⌬ fi < s Ý

f Ž x i . y f Ž x iy1 . s f Ž b . y f Ž a . .

Hence, if M is any number greater than or equal to f Ž b . y f Ž a., then Ý nis1 < ⌬ f i < F M for any partition P of w a, b x. Another example of a function of bounded variation is given in the next theorem. Theorem 6.3.3. If f Ž x . is continuous on w a, b x and its derivative f ⬘Ž x . exists and is bounded on Ž a, b ., then f Ž x . is of bounded variation on w a, b x. Proof. Let Ps
x 0 , x 1 , . . . , x n4 be a partition of w a, b x. By applying the mean value theorem ŽTheorem 4.2.2. on each w x iy1 , x i x, i s 1, 2, . . . , n, we obtain n

Ý

is1

< ⌬ fi < s

n

Ý

f ⬘Ž ␰ i . ⌬ x i

is1 n

FK

Ý Ž x i y x iy1 . s K Ž by a. ,

is1

SOME CLASSES OF FUNCTIONS THAT ARE RIEMANN INTEGRABLE

213

where x iy1 - ␰ i - x i , i s 1, 2, . . . , n, and K ) 0 is such that < f ⬘Ž x .< F K on Ž a, b .. I It should be noted that any function of bounded variation on w a, b x is also bounded there. This is true because if a- x- b, then Ps
a, x, b4 is a partition of w a, b x. Hence, f Ž x . y f Ž a . q f Ž b . y f Ž x . F M. for some positive number M. This implies that < f Ž x .< is bounded on w a, b x since f Ž x . F 12

f Ž x . y f Ž a. q f Ž x . y f Ž b . q f Ž a. q f Ž b .

F 12 Mq f Ž a . q f Ž b . . The converse of this result, however, is not necessarily true, that is, if f Ž x . is bounded, then it may not be of bounded variation. For example, the function

° ¢0,

x cos ž , f Ž x . s~ 2x /

␲

0 - xF 1, xs 0

is bounded on w0, 1x, but is not of bounded variation there. It can be shown that for the partition

½

Ps 0,

1

,

1

2n 2ny1

,...,

5

1 1 , ,1 , 3 2

n < Ý2is1 ⌬ f i < ™ ⬁ as n ™ ⬁ and hence cannot be bounded by a constant M for all n Žsee Exercise 6.4..

Theorem 6.3.4. If f Ž x . is of bounded variation on w a, b x, then it is Riemann integrable there. Proof. Let ⑀ ) 0 be given, and let Ps
x 0 , x 1 , . . . , x n4 be a partition of w a, b x. Then US P Ž f . y LSP Ž f . s

n

Ý Ž Mi y m i . ⌬ x i ,

Ž 6.9 .

is1

where m i and Mi are the infimum and supremum of f Ž x . on w x iy1 , x i x, respectively, i s 1, 2, . . . , n. By the properties of m i and Mi , there exist ␰ i

214

INTEGRATION

and ␩i in w x iy1 , x i x such that for i s 1, 2, . . . , n, m i F f Ž ␰ i . - m i q ⑀ ⬘, Mi y ⑀ ⬘ - f Ž ␩i . F Mi , where ⑀ ⬘ is a small positive number to be determined later. It follows that Mi y m i y 2 ⑀ ⬘ - f Ž ␩i . y f Ž ␰ i . F Mi y m i ,

i s 1, 2, . . . , n.

Hence, Mi y m i - 2 ⑀ ⬘ q f Ž ␩i . y f Ž ␰ i . F 2 ⑀ ⬘ q f Ž ␰ i . y f Ž ␩i . ,

i s 1, 2, . . . , n.

From formula Ž6.9. we obtain n

US P Ž f . y LSP Ž f . - 2 ⑀ ⬘ Ý ⌬ x i q is1

n

Ý f Ž ␰ i . y f Ž ␩i .

⌬ xi .

Ž 6.10 .

,

Ž 6.11 .

is1

Now, if ⌬ p is the norm of P, then n

Ý f Ž ␰ i . y f Ž ␩i .

n

⌬ xi F ⌬ p

is1

Ý f Ž ␰ i . y f Ž ␩i .

is1 m

F⌬p

Ý f Ž z i . y f Ž z iy1 .

is1

where
z 0 , z1 , . . . , z m 4 is a partition Q of w a, b x, which consists of the points x 0 , x 1 , . . . , x n as well as the points ␰ 1 , ␩1 , ␰ 2 , ␩ 2 , . . . , ␰ n , ␩n , that is, Q is a refinement of P obtained by adding the ␰ i ’s and ␩i ’s Ž i s 1, 2, . . . , n.. Since f Ž x . is of bounded variation on w a, b x, there exists a number M) 0 such that m

Ý f Ž z i . y f Ž z iy1 .

F M.

Ž 6.12 .

is1

From Ž6.10., Ž6.11., and Ž6.12. it follows that US P Ž f . y LSP Ž f . - 2 ⑀ ⬘ Ž by a . q M⌬ p .

Ž 6.13 .

Let us now select the partition P such that ⌬ p - ␦ , where M␦ - ⑀r2. If we also choose ⑀ ⬘ such that 2 ⑀ ⬘Ž by a. - ⑀r2, then from Ž6.13. we obtain US P Ž f . y LSP Ž f . - ⑀ . The function f Ž x . is therefore Riemann integrable on w a, b x by Theorem 6.2.1. I

215

PROPERTIES OF THE RIEMANN INTEGRAL

6.4. PROPERTIES OF THE RIEMANN INTEGRAL The Riemann integral has several properties that are useful at both the theoretical and practical levels. Most of these properties are fairly simple and striaghtforward. We shall therefore not prove every one of them in this section. Theorem 6.4.1. If f Ž x . and g Ž x . are Riemann integrable on w a, b x and if c1 and c2 are constants, then c1 f Ž x . q c 2 g Ž x . is Riemann integrable on w a, b x, and

Ha

b

c1 f Ž x . q c 2 g Ž x . dxs c1

Ha f Ž x . dxq c Ha g Ž x . dx. b

b

2

Theorem 6.4.2. If f Ž x . is Riemann integrable on w a, b x, and m F f Ž x . F M for all x in w a, b x, then m Ž b y a. F

Ha f Ž x . dxF M Ž by a. . b

Theorem 6.4.3. If f Ž x . and g Ž x . are Riemann integrable on w a, b x, and if f Ž x . F g Ž x . for all x in w a, b x, then

Ha f Ž x . dxFHa g Ž x . dx. b

Theorem 6.4.4. then

b

If f Ž x . is Riemann integrable on w a, b x and if a- c - b,

Ha f Ž x . dxsHa f Ž x . dxqHc f Ž x . dx. c

b

Theorem 6.4.5. and

b

If f Ž x . is Riemann integrable on w a, b x, then so is < f Ž x .<

Ha f Ž x . dx FHa b

b

f Ž x . dx.

Proof. Let Ps
x 0 , x 1 , . . . , x n4 be a partition of w a, b x. Let m i and Mi be the infimum and supremum of f Ž x ., respectively, on w x iy1 , x i x; and let mXi , MiX be the same for < f Ž x .< . We claim that Mi y m i G MiX y mXi ,

i s 1, 2, . . . , n.

It is obvious that Mi y m i s MiX y mXi if f Ž x . is either nonnegative or nonpositive for all x in w x iy1 , x i x, i s 1, 2, . . . , n. Let us therefore suppose that f Ž x . is

216

INTEGRATION

q y q negative on Dy i and nonnegative on Di , where Di and Di are such that y q Di j Di s Di s w x iy1 , x i x for i s 1, 2, . . . , n. We than have

Mi y m i s sup f Ž x . y inf f Ž x. y Diq

Di

s sup f Ž x . y inf Žy f Ž x . y Diq

Di

.

s sup f Ž x . q sup f Ž x . Diq

Diy

G sup f Ž x . s MiX , Di

since sup D i < f Ž x .< s max
sup D qi < f Ž x .< , sup D yi < f Ž x .< 4 . Hence, Mi y m i G MiX G MiX y mXi for i s 1, 2, . . . , n, which proves our claim. US P Ž < f < . y LSP Ž < f < . s

n

n

is1

is1

Ý Ž MiX y mXi . ⌬ x i F Ý Ž Mi y m i . ⌬ x i .

Hence, US P Ž < f < . y LSP Ž < f < . F US P Ž f . y LSP Ž f . .

Ž 6.14 .

Since f Ž x . is Riemann integrable, the right-hand side of inequality Ž6.14. can be made smaller than any given ⑀ ) 0 by a proper choice of the norm ⌬ p of P. It follows that < f Ž x .< is Riemann integrable on w a, b x by Theorem 6.2.1. Furthermore, since .f Ž x . F < f Ž x .< for all x in w a, b x, then Hab . f Ž x . dxF Hab < f Ž x .< by Theorem 6.4.3, that is,

Ha f Ž x . dxFHa

.

b

Thus, < Hab f Ž x . dx < F Hab < f Ž x .< dx. Corollary 6.4.1.

b

f Ž x . dx.

I

If f Ž x . is Riemann integrable on w a, b x, then so is f 2 Ž x ..

Proof. Using the same notation as in the proof of Theorem 6.4.5, we have that mXi2 and MiX 2 are, respectively, the infimum and supremum of f 2 Ž x . on w x iy1 , x i x for i s 1, 2, . . . , n. Now, MiX 2 y mXi2 s Ž MiX y mXi . Ž MiX q mXi . F 2 M⬘ Ž MiX y mXi . F 2 M⬘ Ž Mi y m i . ,

i s 1, 2, . . . , n,

Ž 6.15 .

217

PROPERTIES OF THE RIEMANN INTEGRAL

where M⬘ is the supremum of < f Ž x .< on w a, b x. The Riemann integrability of f 2 Ž x . now follows from inequality Ž6.15. by the Riemann integrability of f Ž x .. I Corollary 6.4.2. If f Ž x . and g Ž x . are Riemann integrable on w a, b x, then so is their product f Ž x . g Ž x .. Proof. This follows directly from the identity 4 f Ž x. g Ž x. s f Ž x. qg Ž x.

2

y f Ž x. yg Ž x. , 2

Ž 6.16 .

and the fact that the squares of f Ž x . q g Ž x . and f Ž x . y g Ž x . are Riemann integrable on w a, b x by Theorem 6.4.1 and Corollary 6.4.1. I Theorem 6.4.6 ŽThe Mean Value Theorem for Integrals .. If f Ž x . is continuous on w a, b x, then there exists a point c g w a, b x such that

Ha f Ž x . dxs Ž by a. f Ž c . . b

Proof. By Theorem 6.4.2 we have mF

1

Ha f Ž x . dxF M, b

by a

where m and M are, respectively, the infimum and supremum of f Ž x . on w a, b x. Since f Ž x . is continuous, then by Corollary 3.4.1 it must attain the values m and M at some points inside w a, b x. Furthermore, by the intermediate-value theorem ŽTheorem 3.4.4., f Ž x . assumes every value between m and M. Hence, there is a point c g w a, b x such that f Ž c. s Definition 6.4.1.

1

Ha f Ž x . dx. b

by a

I

Let f Ž x . be Riemann integrable on w a, b x. The function FŽ x. s

Ha f Ž t . dt, x

is called an indefinite integral of f Ž x ..

aF x F b, I

Theorem 6.4.7. If f Ž x . is Riemann integrable on w a, b x, then F Ž x . s dt is uniformly continuous on w a, b x.

Hax f Ž t .

218

INTEGRATION

Proof. Let x 1 , x 2 be in w a, b x, x 1 - x 2 . Then, F Ž x 2 . y F Ž x1 . s

Ha

x2

f Ž t . dty

s

Hx

x2

f Ž t . dt ,

by Theorem 6.4.4

f Ž t . dt,

by Theorem 6.4.5

F

Hx

Ha

x1

f Ž t . dt

1

x2 1

F M⬘ Ž x 2 y x 1 . , where M⬘ is the supremum of < f Ž x .< on w a, b x. Thus if ⑀ ) 0 is given, then < F Ž x 2 . y F Ž x 1 .< - ⑀ provided that < x 1 y x 2 < - ⑀rM⬘. This proves uniform continuity of F Ž x . on w a, b x. I The next theorem presents a practical way for evaluating the Riemann integral on w a, b x. Theorem 6.4.8. Suppose that f Ž x . is continuous on w a, b x. Let F Ž x . s Hax f Ž t . dt. Then we have the following: i. dF Ž x .rdxs f Ž x ., aF xF b. ii. Hab f Ž x . dxs G Ž b . y GŽ a., where GŽ x . s F Ž x . q c, and c is an arbitrary constant. Proof. We have dF Ž x . dx

s

d

1

lim H f Ž t . dts h™0 H dx a h a x

1

H h™0 h x

s lim

xqh

f Ž t . dt,

xqh

f Ž t . dty

Ha f Ž t . dt x

by Theorem 6.4.4

s lim f Ž xq ␪ h . , h™0

by Theorem 6.4.6, where 0 F ␪ F 1. Hence, dF Ž x . dx

s lim f Ž xq ␪ h . s f Ž x . h™0

by the continuity of f Ž x .. This result indicates that an indefinite integral of f Ž x . is any function whose derivative is equal to f Ž x .. It is therefore unique up to a constant. Thus both F Ž x . and F Ž x . q c, where c is an arbitrary constant, are considered to be indefinite integrals.

219

PROPERTIES OF THE RIEMANN INTEGRAL

To prove the second part of the theorem, let G Ž x . be defined on w a, b x as GŽ x. sF Ž x. qcs

Ha f Ž t . dtq c, x

that is, GŽ x . is an indefinite integral of f Ž x .. If xs a, then G Ž a. s c, since F Ž a. s 0. Also, if xs b, then G Ž b . s F Ž b . q c s Hab f Ž t . dtq G Ž a.. It follows that

Ha f Ž t . dts G Ž b . y G Ž a. . b

I

This result is known as the fundamental theorem of calculus. It is generally attributed to Isaac Barrow Ž1630᎐1677., who was the first to realize that differentiation and integration are inverse operations. One advantage of this theorem is that it provides a practical way to evaluate the integral of f Ž x . on w a, b x. 6.4.1. Change of Variables in Riemann Integration There are situations in which the variable x in a Riemann integral is a function of some other variable, say u. In this case, it may be of interest to determine how the integral can be expressed and evaluated under the given transformation. One advantage of this change of variable is the possibility of simplifying the actual evaluation of the integral, provided that the transformation is properly chosen. Theorem 6.4.9. Let f Ž x . be continuous on w ␣ , ␤ x, and let xs g Ž u. be a function whose derivative g ⬘Ž u. exists and is continuous on w c, d x. Suppose that the range of g is contained inside w ␣ , ␤ x. If a, b are points in w ␣ , ␤ x such that as g Ž c . and bs g Ž d ., then

Ha f Ž x . dxsHc f b

d

g Ž u . g ⬘ Ž u . du.

Proof. Let F Ž x . s Hax f Ž t . dt. By Theorem 6.4.8, F⬘Ž x . s f Ž x .. Let G Ž u. be defined as G Ž u. s

Hc f u

g Ž t . g ⬘ Ž t . dt.

Since f, g, and g ⬘ are continuous, then by Theorem 6.4.8 we have dG Ž u . du

s f g Ž u. g ⬘Ž u. .

Ž 6.17 .

220

INTEGRATION

However, according to the chain rule ŽTheorem 4.1.3., dF g Ž u . du

s

dF g Ž u .

dg Ž u .

dg Ž u .

du

s f g Ž u. g ⬘Ž u. .

Ž 6.18 .

From formulas Ž6.17. and Ž6.18. we conclude that G Ž u. y F g Ž u. s ␭,

Ž 6.19 .

where ␭ is a constant. If a and b are points in w ␣ , ␤ x such that as g Ž c ., bs g Ž d ., then when u s c, we have G Ž c . s 0 and ␭ s yF w g Ž c .x s yF Ž a. s 0. Furthermore, when u s d, G Ž d . s Hcd f w g Ž t .x g ⬘Ž t . dt. From Ž6.19. we then obtain GŽ d. s

Hc f d

g Ž t . g ⬘ Ž t . dts F g Ž d . q ␭

s F Ž b. s

Ha f Ž x . dx. b

For example, consider the integral H12 Ž2 t 2 y 1.1r2 t dt. Let xs 2 t 2 y 1. Then dxs 4 t dt, and by Theorem 6.4.9,

H1 Ž 2 t 2

2

y 1.

1r2

t dts

1

Hx 4 1

7 1r2

dx.

An indefinite integral of x 1r2 is given by 23 x 3r2 . Hence,

H1 Ž 2 t 2

2

y 1.

1r2

t dts 14 Ž 23 . Ž 7 3r2 y 1 . s 16 Ž 7 3r2 y 1 . .

I

6.5. IMPROPER RIEMANN INTEGRALS In our study of the Riemann integral we have only considered integrals of functions that are bounded on a finite interval w a, b x. We now extend the scope of Riemann integration to include situations where the integrand can become unbounded at one or more points inside the range of integration, which can also be infinite. In such situations, the Riemann integral is called an improper integral. There are two kinds of improper integrals. If f Ž x . is Riemann integrable on w a, b x for any b) a, then Ha⬁ f Ž x . dx is called an improper integral of the first kind, where the range of integration is infinite. If, however, f Ž x .

221

IMPROPER RIEMANN INTEGRALS

becomes infinite at a finite number of points inside the range of integration, then the integral Hab f Ž x . dx is said to be improper of the second kind. Definition 6.5.1. Let F Ž z . s Haz f Ž x . dx. Suppose that F Ž z . exists for any value of z greater than a. If F Ž z . has a finite limit L as z™ ⬁, then the improper integral Ha⬁ f Ž x . dx is said to converge to L. In this case, L represents the Riemann integral of f Ž x . on w a, ⬁. and we write Ls

⬁

Ha

f Ž x . dx.

On the other hand, if L s "⬁, then the improper integral Ha⬁ f Ž x . dx is said a to diverge. By the same token, we can define the integral Hy⬁ f Ž x . dx as the ⬁ a limit, if it exists, of Hyz f Ž x . dx as z™ ⬁. Also, Hy⬁ f Ž x . dx is defined as ⬁

lim H f Ž x . dxq lim H f Ž x . dx, Hy⬁ f Ž x . dxs u™⬁ z™⬁ a yu a

z

where a is any finite number, provided that both limits exist. The convergence of Ha⬁ f Ž x . dx can be determined by using the Cauchy criterion in a manner similar to the one used in the study of convergence of sequences Žsee Section 5.1.1.. Theorem 6.5.1. The improper integral Ha⬁ f Ž x . dx converges if and only if for a given ⑀ ) 0 there exists a z 0 such that

Hz

z2

f Ž x . dx - ⑀ ,

Ž 6.20 .

1

whenever z1 and z 2 exceed z 0 . Proof. If F Ž z . s Haz f Ž x . dx has a limit L as z ™ ⬁, then for a given ⑀ ) 0 there exists z 0 such that for z ) z 0 . FŽ z. yL -

⑀ 2

.

Now, if both z1 and z 2 exceed z 0 , then

Hz

z2

f Ž x . dx s F Ž z 2 . y F Ž z1 .

1

F F Ž z 2 . y L q F Ž z1 . y L - ⑀ . Vice versa, if condition Ž6.20. is satisfied, then we need to show that F Ž z . has a limit as z ™ ⬁. Let us therefore define the sequence
g n4⬁ns1 , where g n is

222

INTEGRATION

given by gn s

Ha

aqn

f Ž x . dx,

n s 1, 2, . . . .

It follows that for any ⑀ ) 0, < gn y gm < s

Haqm f Ž x . dx aqn

-⑀ ,

if m and n are large enough. This implies that
g n4⬁ns1 is a Cauchy sequence; hence it converges by Theorem 5.1.6. Let g s lim n™⬁ g n . To show that lim z ™⬁ F Ž z . s g, let us write F Ž z . y g s F Ž z . y gn q gn y g F F Ž z . y gn q < gn y g < .

Ž 6.21 .

Suppose ⑀ ) 0 is given. There exists an integer N1 such that < g n y g < - ⑀r2 if n ) N1. Also, there exists an integer N2 such that F Ž z . y gn s

⑀

Haqn f Ž x . dx - 2 z

Ž 6.22 .

if z) aq n ) N2 . Thus by choosing z ) aq n, where n ) maxŽ N1 , N2 y a., we get from inequalities Ž6.21. and Ž6.22. FŽ z. yg -⑀ . This completes the proof.

I

Definition 6.5.2. If the improper integral Ha⬁ < f Ž x .< dx is convergent, then the integral Ha⬁ f Ž x . dx is said to be absolutely convergent. If Ha⬁ f Ž x . dx is convergent but not absolutely, then it is said to be conditionally convergent. I It is easy to show that an improper integral is convergent if it converges absolutely. As with the case of series of positive terms, there are comparison tests that can be used to test for convergence of improper integrals of the first kind of nonnegative functions. These tests are described in the following theorems. Theorem 6.5.2. Let f Ž x . be a nonnegative function that is Riemann integrable on w a, b x for every bG a. Suppose that there exists a function g Ž x . such that f Ž x . F g Ž x . for xG a. If Ha⬁ g Ž x . dx converges, then so does Ha⬁ f Ž x . dx

223

IMPROPER RIEMANN INTEGRALS

and we have ⬁

Ha

f Ž x . dxF

⬁

Ha g Ž x . dx.

I

Proof. See Exercise 6.7.

Theorem 6.5.3. Let f Ž x . and g Ž x . be nonnegative functions that are Riemann integrable on w a, b x for every bG a. If f Ž x.

lim

sk,

gŽ x.

x™⬁

where k is a positive constant, then Ha⬁ f Ž x . dx and Ha⬁ g Ž x . dx are either both convergent or both divergent. I

Proof. See Exercise 6.8.

EXAMPLE 6.5.1. Consider the integral H1⬁eyx x 2 dx. We have that e x s 1 q Hence, for xG 1, e x ) x prp!, where p is any positive integer. If p is chosen such that py 2 G 2, then

Ý⬁ns1 Ž x nrn!..

eyx x 2 -

p! x

py2

F

p!

.

x2

However, H1⬁Ž dxrx 2 . s wy1rx x⬁1 s 1. Therefore, by Theorem 6.5.2, the integral of eyx x 2 on w1, ⬁. is convergent. EXAMPLE 6.5.2. gent, since

The integral H0⬁wŽsin x .rŽ xq 1. 2 x dx is absolutely conver< sin x <

Ž xq 1 .

2

F

1

Ž xq 1 .

2

and dx

⬁

H0

Ž xq 1 .

2

s y

⬁

1

s 1.

xq 1

0

EXAMPLE 6.5.3. The integral H0⬁Žsin xrx . dx is conditionally convergent. We first show that H0⬁Žsin xrx . dx is convergent. We have that ⬁

H0

sin x x

dxs

H0

1

sin x x

dxq

⬁

H1

sin x x

dx.

Ž 6.23 .

224

INTEGRATION

By Exercise 6.3, Žsin x .rx is Riemann integrable on w0, 1x, since it is continuous there except at xs 0, which is a discontinuity of the first kind Žsee Definition 3.4.2.. As for the second integral in Ž6.23., we have for z 2 ) z1 ) 1,

Hz

z2

sin x x

1

dxs y

cos x

z2

x

z1

cos z1

s

z1

y

y

Hz

cos z 2 z2

z2

cos x

dx

x2

1

y

Hz

z2

cos x

dx.

x2

1

Thus

Hz

z2 1

sin x x

dx F

1 z1

q

1 z2

q

Hz

z2 1

dx x

2

s

2 z1

.

Since 2rz1 can be made arbitrarily small by choosing z1 large enough, then by Theorem 6.5.1, H1⬁Žsin xrx . dx is convergent and so is H0⬁Žsin xrx . dx. It remains to show that H0⬁Žsin xrx . dx is not absolutely convergent. This follows from the fact that Žsee Exercise 6.10. sin x

n␲

H n™⬁ 0 lim

x

dxs ⬁.

Convergence of improper integrals of the first kind can be used to determine convergence of series of positive terms Žsee Section 5.2.1.. This is based on the next theorem. Theorem 6.5.4 ŽMaclaurin’s Integral Test.. Let Ý⬁ns1 a n be a series of positive terms such that a nq1 F a n for n G 1. Let f Ž x . be a positive nonincreasing function defined on w1, ⬁. such that f Ž n. s a n , n s 1, 2, . . . , and f Ž x . ™ 0 as x™ ⬁. Then, Ý⬁ns1 a n converges if and only if the improper integral H1⬁ f Ž x . dx converges. Proof. If n G 1 and n F xF n q 1, then a n s f Ž n . G f Ž x . G f Ž n q 1 . s a nq1 . By Theorem 6.4.2 we have for n G 1 an G

Hn

nq1

f Ž x . dxG a nq1 .

Ž 6.24 .

If sn s Ý nks1 a k is the nth partial sum of the series, then from inequality Ž6.24. we obtain sn G

H1

nq1

f Ž x . dxG snq1 y a1 .

Ž 6.25 .

225

IMPROPER RIEMANN INTEGRALS

If the series Ý⬁ns1 a n converges to the sum s, then s G sn for all n. Consequently, the sequence whose nth term is F Ž n q 1. s H1nq1 f Ž x . dx is monotone increasing and is bounded by s; hence it must have a limit. Therefore, the integral H1⬁ f Ž x . dx converges. Now, let us suppose that H1⬁ f Ž x . dx is convergent and is equal to L. Then from inequality Ž6.25. we obtain snq1 F a1 q

H1

nq1

f Ž x . dxF a1 q L,

n G 1,

Ž 6.26 .

since f Ž x . is positive. Inequality Ž6.26. indicates that the monotone increasing sequence
sn4⬁ns1 is bounded hence it has a limit, which is the sum of the series. I Theorem 6.5.4 provides a test of convergence for a series of positive terms. Of course, the usefulness of this test depends on how easy it is to integrate the function f Ž x .. As an example of using the integral test, consider the harmonic series Ý⬁ns1 Ž1rn.. If f Ž x . is defined as f Ž x . s 1rx, xG 1, then F Ž x . s H1x f Ž t . dts log x. Since F Ž x . goes to infinity as x™ ⬁, the harmonic series must therefore be divergent, as was shown in Chapter 5. On the other hand, the series Ý⬁ns1 Ž1rn2 . is convergent, since F Ž x . s H1x Ž dtrt 2 . s 1 y 1rx, which converges to 1 as x™ ⬁. 6.5.1. Improper Riemann Integrals of the Second Kind Let us now consider integrals of the form Hab f Ž x . dx where w a, b x is a finite interval and the integrand becomes infinite at a finite number of points inside w a, b x. Such integrals are called improper integrals of the second kind. Suppose, for example, that f Ž x . ™ ⬁ as x™ aq. Then Hab f Ž x . dx is said to converge if the limit limq

⑀™0

Haq⑀ f Ž x . dx b

exists and is finite. Similarly, if f Ž x . ™ ⬁ as x™ by, then Hab f Ž x . dx is convergent if the limit limq

⑀™0

Ha

by⑀

f Ž x . dx

exists. Furthermore, if f Ž x . ™ ⬁ as x™ c, where a- c - b, then Hab f Ž x . dx is the sum of Hac f Ž x . dx and Hcb f Ž x . dx provided that both integrals converge. By definition, if f Ž x . ™ ⬁ as x™ x 0 , where x 0 g w a, b x, then x 0 is said to be a singularity of f Ž x ..

226

INTEGRATION

The following theorems can help in determining convergence of integrals of the second kind. They are similar to Theorems 6.5.1, 6.5.2, and 6.5.3. Their proofs will therefore be omitted. Theorem 6.5.5. If f Ž x . ™ ⬁ as x™ aq, then Hab f Ž x . dx converges if and only if for a given ⑀ ) 0 there exists a z 0 such that

Hz

z2

f Ž x . dx - ⑀ ,

1

where z1 and z 2 are any two numbers such that a- z1 - z 2 - z 0 - b. Theorem 6.5.6. Let f Ž x . be a nonnegative function such that Hcb f Ž x . dx exists for every c in Ž a, b x. If there exists a function g Ž x . such that f Ž x . F g Ž x . for all x in Ž a, b x, and if Hcb g Ž x . dx converges as c ™ aq, then so does Hcb f Ž x . dx and we have

Ha f Ž x . dxFHa g Ž x . dx. b

b

Theorem 6.5.7. Let f Ž x . and g Ž x . be nonnegative functions that are Riemann integrable on w c, b x for every c such that a- c F b. If limq

x™a

f Ž x. gŽ x.

s k,

where k is a positive constant, then Hab f Ž x . dx and Hab g Ž x . dx are either both convergent or both divergent. Definition 6.5.3. Let Hab f Ž x . dx be an improper integral of the second kind. If Hab < f Ž x .< dx converges, then Hab f Ž x . dx is said to converge absolutely. If, however, Hab f Ž x . dx is convergent, but not absolutely, then it is said to be conditionally convergent. I Theorem 6.5.8.

If Hab < f Ž x .< dx converges, then so does Hab f Ž x . dx.

EXAMPLE 6.5.4. Consider the integral H01 eyx x ny1 dx, where n ) 0. If 0 - n - 1, then the integral is improper of the second kind, since x ny1 ™ ⬁ as x ™ 0q. Thus, xs 0 is a singularity of the integrand. Since lim

x™0 q

eyx x ny1 x ny1

s 1,

then the behavior of H01 eyx x ny1 dx with regard to convergence or divergence is the same as that of H01 x ny1 dx. But H01 x ny1 dxs Ž1rn.w x n x10 s 1rn is convergent, and so is H01 eyx x ny1 dx.

227

CONVERGENCE OF A SEQUENCE OF RIEMANN INTEGRALS

EXAMPLE 6.5.5. H01 Žsin xrx 2 . dx. The integrand has a singularity at xs 0. Let g Ž x . s 1rx. Then, Žsin x .rw x 2 g Ž x .x ™ 1 as x™ 0q. But H01 Ž dxrx . s wlog x x10 is divergent, since log x™ y⬁ as x™ 0q. Therefore, H01 Žsin xrx 2 . dx is divergent. EXAMPLE 6.5.6. Consider the integral H02 Ž x 2 y 3 xq 1.rw x Ž xy 1. 2 x dx. Here, the integrand has two singularities, namely x s 0 and x s 1, inside w0, 2x. We can therefore write

H0

2

x 2 y 3 xq 1 x Ž xy 1 .

2

dxs limq t™0

Ht

1r2

x2 y3 xq1 x Ž xy 1 .

q limy

H1r2

q limq

H®

u™1

®™1

u

2

dx

2

x 2 y 3 xq 1 x Ž x y 1.

x 2 y 3 xq 1 x Ž xy 1 .

2

dx

2

dx.

We note that x 2 y 3 xq 1 x Ž x y 1.

2

s

1 x

y

1

Ž xy 1 .

.

2

Hence,

H0

2

x 2 y 3 xq 1 x Ž x y 1.

2

dxs limq log xq t™0

xy 1

q limy log x q u™1

q limq log xq ®™1

1r2

1

t

1 xy 1 1 xy 1

u

1r2 2

. ®

None of the above limits exists as a finite number. This integral is therefore divergent.

6.6. CONVERGENCE OF A SEQUENCE OF RIEMANN INTEGRALS In the present section we confine our attention to the limiting behavior of integrals of a sequence of functions
f nŽ x .4⬁ns1 .

228

INTEGRATION

Theorem 6.6.1. Suppose that f nŽ x . is Riemann integrable on w a, b x for n G 1. If f nŽ x . converges uniformly to f Ž x . on w a, b x as n ™ ⬁, then f Ž x . is Riemann integrable on w a, b x and

H f Ž x . dxsHa f Ž x . dx. n™⬁ a lim

b

b

n

Proof. Let us first show that f Ž x . is Riemann integrable on w a, b x. Let ⑀ ) 0 be given. Since f nŽ x . converges uniformly to f Ž x ., then there exists an integer n 0 that depends only on ⑀ such that fn Ž x . y f Ž x . -

⑀

Ž 6.27 .

3 Ž by a .

if n ) n 0 for all xg w a, b x. Let n1 ) n 0 . Since f n1Ž x . is Riemann integrable on w a, b x, then by Theorem 6.2.1 there exists a ␦ ) 0 such that US P Ž f n1 . y LSP Ž f n1 . -

⑀

Ž 6.28 .

3

for any partition P of w a, b x with a norm ⌬ p - ␦ . Now, from inequality Ž6.27. we have f Ž x . - f n1Ž x . q f Ž x . ) f n1Ž x . y

⑀

,

3 Ž by a .

⑀

.

3 Ž by a .

We conclude that US P Ž f . F US P Ž f n1 . q LSP Ž f . G LSP Ž f n1 . y

⑀ 3

⑀ 3

,

Ž 6.29 .

.

Ž 6.30 .

From inequalities Ž6.28., Ž6.29., and Ž6.30. it follows that if ⌬ p - ␦ , then US P Ž f . y LSP Ž f . F US P Ž f n1 . y LSP Ž f n1 . q

2⑀ 3

-⑀ .

Ž 6.31 .

Inequality Ž6.31. shows that f Ž x . is Riemann integrable on w a, b x, again by Theorem 6.2.1.

229

SOME FUNDAMENTAL INEQUALITIES

Let us now show that lim

n™⬁

Ha f Ž x . dxsHa f Ž x . dx. b

b

Ž 6.32 .

n

From inequality Ž6.27. we have for n ) n 0 ,

Ha f Ž x . dxyHa f Ž x . dx FHa b

b

b

f n Ž x . y f Ž x . dx

n

-

⑀ 3

,

and the result follows, since ⑀ is an arbitrary positive number.

I

6.7. SOME FUNDAMENTAL INEQUALITIES In this section we consider certain well-known inequalities for the Riemann integral. 6.7.1. The Cauchy–Schwarz Inequality Theorem 6.7.1. Suppose that f Ž x . and g Ž x . are such that f 2 Ž x . and g x . are Riemann integrable on w a, b x. Then 2Ž

Ha f Ž x . g Ž x . dx b

2

Ha f

b 2

F

Ž x . dx

Ha g

b 2

Ž x . dx .

Ž 6.33 .

The limits of integration may be finite or infinite. Proof. Let c1 and c 2 be constants, not both zero. Without loss of generality, let us assume that c 2 / 0. Then

Ha

b

c1 f Ž x . q c 2 g Ž x .

2

dxG 0.

Thus the quadratic form c12

Ha f

b 2

Ž x . dxq 2 c1 c2H f Ž x . g Ž x . dxq c22H g 2 Ž x . dx b

a

b

a

230

INTEGRATION

is nonnegative for all c1 and c 2 . It follows that its discriminant, namely, c 22

Ha

b

2

f Ž x . g Ž x . dx

y c 22

Ha f

b 2

Ž x . dx

Ha g

b 2

Ž x . dx

must be nonpositive, that is,

Ha

b

2

f Ž x . g Ž x . dx

F

Ha f

b 2

Ha g

b 2

Ž x . dx

Ž x . dx .

It is easy to see that if f Ž x . and g Ž x . are linearly related wthat is, there exist constants ␶ 1 and ␶ 2 , not both zero, such that ␶ 1 f Ž x . q ␶ 2 g Ž x . s 0x, then inequality Ž6.33. becomes an equality. I 6.7.2. Holder’s Inequality ¨ This is a generalization of the Cauchy᎐Schwarz inequality due to Otto Ž1859᎐1937.. To prove Holder’s Holder inequality we need the following ¨ ¨ lemmas: Lemma 6.7.1. Let a1 , a1 , . . . , a n ; ␭1 , ␭2 , . . . , ␭ n be nonnegative numbers such that Ý nis1 ␭ i s 1. Then n

Ł ai␭i F

is1

n

Ý ␭i ai .

Ž 6.34 .

is1

The right-hand side of inequality Ž6.34. is a weighted arithmetic mean of the a i ’s, and the left-hand side is a weighted geometric mean. Proof. This lemma is an extension of a result given in Section 3.7 concerning the properties of convex functions Žsee Exercise 6.19.. I Lemma 6.7.2. Suppose that f 1Ž x ., f 2 Ž x ., . . . , f nŽ x . are nonnegative and Riemann integrable on w a, b x. If ␭1 , ␭2 , . . . , ␭ n are nonnegative numbers such that Ý nis1 ␭ i s 1, then n

Ha is1 Łf b

␭i i

n

Ž x . dxF Ł is1

Ha f Ž x . dx b

i

␭i

.

Ž 6.35 .

Proof. Without loss of generality, let us assume that Hab f i Ž x . dx) 0 for i s 1, 2, . . . , n winequality Ž6.35. is obviously true if at least one f i Ž x . is

231

SOME FUNDAMENTAL INEQUALITIES

identically equal to zerox. By Lemma 6.7.1 we have n Hab Ł is1 f i␭ i Ž x . dx n Ł is1 Hab f i Ž x . dx

␭i

␭1

f 1Ž x .

s

Ha

F

Ha Ý

b

Hab f 1 Ž x . dx

is1

␭n

fn Ž x .

⭈⭈⭈

Hab f 2 Ž x . dx

␭i fi Ž x .

n

b

␭2

f2 Ž x .

dx

Hab f n Ž x . dx

n

Hab f i Ž x . dx

dxs

Ý ␭ i s 1.

is1

Hence, inequality Ž6.35. follows.

I

Theorem 6.7.2 ŽHolder’s Inequality.. Let p and q be two positive num¨ bers such that 1rpq 1rqs 1. If < f Ž x .< p and < g Ž x .< q are Riemann integrable on w a, b x, then

Ha

b

Ha

b

f Ž x . g Ž x . dx F

f Ž x.

p

1rp

dx

Ha

b

gŽ x.

q

1rq

dx

.

Proof. Define the functions p

q

uŽ x . s f Ž x . ,

®Ž x . s g Ž x . .

Then, by Lemma 6.7.2,

Ha u Ž x . b

1rp

®Ž x .

1rq

dxF

Ha u Ž x . dx b

1rp

Ha ® Ž x . dx b

1rq

,

that is,

Ha

b

f Ž x . g Ž x . dxF

Ha

b

p

f Ž x.

1rp

dx

Ha

b

gŽ x.

q

1rq

dx

.

Ž 6.36 .

The theorem follows from inequality Ž6.36. and the fact that

Ha f Ž x . g Ž x . dx FHa b

b

f Ž x . g Ž x . dx.

We note that the Cauchy-Schwarz inequality can be deduced from Theorem 6.7.2 by taking ps q s 2. I

232

INTEGRATION

6.7.3. Minkowski’s Inequality The following inequality is due to Hermann Minkowski Ž1864᎐1909.. Theorem 6.7.3. Suppose that f Ž x . and g Ž x . are functions such that < f Ž x .< p and < g Ž x .< p are Riemann integrable on w a, b x, where 1 F p- ⬁. Then

Ha

b

1rp

p

f Ž x. qg Ž x.

dx

F

Ha

b

1rp

p

f Ž x.

Ha

b

q

dx

p

gŽ x.

1rp

dx

.

Proof. The theorem is obviously true if ps 1 by the triangle inequality. We therefore assume that p) 1. Let q be a positive number such that 1rpq 1rqs 1. Hence, ps pŽ1rpq 1rq . s 1 q prq. Let us now write p

f Ž x. qg Ž x. s f Ž x. qg Ž x. F f Ž x.

f Ž x. qg Ž x. prq

f Ž x. qg Ž x.

prq

q gŽ x.

f Ž x. qg Ž x.

prq

.

Ž 6.37 . By applying Holder’s inequality to the two terms on the right-hand side of ¨ inequality Ž6.37. we obtain

Ha

b

f Ž x.

Ha

b

F

Ha

b

gŽ x.

p

f Ž x.

1rp

dx

f Ž x. qg Ž x.

Ha

b

F

prq

f Ž x. qg Ž x.

p

gŽ x.

Ha

b

prq

1rp

dx

dx f Ž x. qg Ž x.

p

f Ž x. qg Ž x.

p

1rq

dx

,

Ž 6.38 .

.

Ž 6.39 .

dx

Ha

b

1rq

dx

From inequalities Ž6.37., Ž6.38., and Ž6.39. we conclude that

Ha

b

f Ž x. qg Ž x.

p

dxF

Ha

b

=

f Ž x. qg Ž x.

½H

b

a

f Ž x.

p

p

1rq

dx 1rp

dx

q

Ha

b

gŽ x.

p

1rp

dx

5

.

Ž 6.40 .

233

SOME FUNDAMENTAL INEQUALITIES

Since 1 y 1rqs 1rp, inequality Ž6.40. can be written as

Ha

b

p

f Ž x. qg Ž x.

1rp

F

dx

Ha

b

f Ž x.

p

1rp

Ha

b

q

dx

gŽ x.

p

1rp

dx

.

Minkowski’s inequality can be extended to integrals involving more than two functions. It can be shown Žsee Exercise 6.20. that if < f i Ž x .< p is Riemann integrable on w a, b x for i s 1, 2, . . . , n, then n

Ha Ý b

1rp

p

fi Ž x .

dx

is1

n

F

Ý

is1

Ha

b

fi Ž x .

p

1rp

dx

I

.

6.7.4. Jensen’s Inequality Theorem 6.7.4. Let X be a random variable with a finite expected value, ␮ s E Ž X .. If ␾ Ž x . is a twice differentiable convex function, then E ␾ Ž X . G␾ EŽ X . . Proof. Since ␾ Ž x . is convex and ␾ ⬙ Ž x . exists, then we must have ␾ ⬙ Ž x . G 0. By applying the mean value theorem ŽTheorem 4.2.2. around ␮ we obtain

␾ Ž X . s ␾ Ž ␮ . q Ž X y ␮ . ␾ ⬘Ž c . , where c is between ␮ and X. If X y ␮ ) 0, then c ) ␮ and hence ␾ ⬘Ž c . G ␾ ⬘Ž ␮ ., since ␾ ⬙ Ž x . is nonnegative. Thus,

␾ Ž X . y ␾ Ž ␮ . s Ž X y ␮ . ␾ ⬘Ž c . G Ž X y ␮ . ␾ ⬘Ž ␮ . .

Ž 6.41 .

On the other hand, if X y ␮ - 0, then c - ␮ and ␾ ⬘Ž c . F ␾ ⬘Ž ␮ .. Hence,

␾ Ž X . y ␾ Ž ␮ . s Ž X y ␮ . ␾ ⬘Ž c . G Ž X y ␮ . ␾ ⬘Ž ␮ . . From inequalities Ž6.41. and Ž6.42. we conclude that E ␾ Ž X . y ␾ Ž ␮ . G ␾ ⬘ Ž ␮ . E Ž X y ␮ . s 0, which implies that E ␾ Ž X . G␾ Ž ␮. , since E w ␾ Ž ␮ .x s ␾ Ž ␮ ..

I

Ž 6.42 .

234

INTEGRATION

6.8. RIEMANN–STIELTJES INTEGRAL In this section we consider a more general integral, namely the Riemann᎐ Stieltjes integral. The concept on which this integral is based can be attributed to a combination of ideas by Georg Friedrich Riemann Ž1826᎐1866. and the Dutch mathematician Thomas Joannes Stieltjes Ž1856᎐1894.. The Riemann-Stieltjes integral involves two functions f Ž x . and g Ž x ., both defined on the interval w a, b x, and is denoted by Hab f Ž x . dg Ž x .. In particular, if g Ž x . s x we obtain the Riemann integral Hab f Ž x . dx. Thus the Riemann integral is a special case of the Riemann-Stieltjes integral. The definition of the Riemann-Stieltjes integral of f Ž x . with respect to g Ž x . on w a, b x is similar to that of the Riemann integral. If f Ž x . is bounded on w a, b x, if g Ž x . is monotone increasing on w a, b x, and if Ps
x 0 , x 1 , . . . , x n4 is a partition of w a, b x, then as in Section 6.2, we define the sums LSP Ž f , g . s

n

Ý mi ⌬ gi ,

is1

US P Ž f , g . s

n

Ý Mi ⌬ g i ,

is1

where m i and Mi are, respectively, the infimum and supremum of f Ž x . on w x iy1 , x i x, ⌬ g i s g Ž x i . y g Ž x iy1 ., i s 1, 2, . . . , n. If for a given ⑀ ) 0 there exists a ␦ ) 0 such that US P Ž f , g . y LSP Ž f , g . - ⑀

Ž 6.43 .

whenever ⌬ p - ␦ , where ⌬ p is the norm of P, then f Ž x . is said to be Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x. In this case,

Ha f Ž x . dg Ž x . s infP US b

P

Ž f , g . s sup LSP Ž f , g . . P

Condition Ž6.43. is both necessary and sufficient for the existence of the Riemann᎐Stieltjes integral. Equivalently, suppose that for a given partition Ps
x 0 , x 1 , . . . , x n4 we define the sum SŽ P , f , g . s

n

Ý f Ž ti . ⌬ g i ,

Ž 6.44 .

is1

where t i is a point in the interval w x iy1 , x i x, i s 1, 2, . . . , n. Then f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x if for any ⑀ ) 0

RIEMANN᎐ STIELTJES INTEGRAL

235

there exists a ␦ ) 0 such that SŽ P , f , g . y

Ha f Ž x . dg Ž x . b

-⑀

Ž 6.45 .

for any partition P of w a, b x with a norm ⌬ p - ␦ , and for any choice of the point t i in w x iy1 , x i x, i s 1, 2, . . . , n. Theorems concerning the Riemann᎐Stieltjes integral are very similar to those seen earlier concerning the Riemann integral. In particular, we have the following theorems: Theorem 6.8.1. If f Ž x . is continuous on w a, b x, then Riemann᎐Stieltjes integrable on w a, b x. Proof. See Exercise 6.21.

f Ž x . is

I

Theorem 6.8.2. If f Ž x . is monotone increasing Žor monotone decreasing . on w a, b x, and g Ž x . is continuous on w a, b x, then f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x. Proof. See Exercise 6.22.

I

The next theorem shows that under certain conditions, the Riemann᎐ Stieltjes integral reduces to the Riemann integral. Theorem 6.8.3. Suppose that f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x, where g Ž x . has a continuous derivative g ⬘Ž x . on w a, b x. Then

Ha f Ž x . dg Ž x . sHa f Ž x . g ⬘ Ž x . dx. b

b

Proof. Let Ps
x 0 , x 1 , . . . , x n4 be a partition of w a, b x. Consider the sum S Ž P , h. s

n

Ý h Ž ti . ⌬ x i ,

Ž 6.46 .

is1

where hŽ x . s f Ž x . g ⬘Ž x . and x iy1 F t i F x i , i s 1, 2, . . . , n. Let us also consider the sum SŽ P , f , g . s

n

Ý f Ž ti . ⌬ g i ,

is1

Ž 6.47 .

236

INTEGRATION

If we apply the mean value theorem ŽTheorem 4.2.2. to g Ž x ., we obtain ⌬ g i s g Ž x i . y g Ž x iy1 . s g ⬘ Ž z i . ⌬ x i ,

i s 1, 2, . . . , n,

Ž 6.48 .

where x iy1 - z i - x i , i s 1, 2, . . . , n. From Ž6.46., Ž6.47., and Ž6.48. we can then write S Ž P , f , g . y S Ž P , h. s

n

Ý f Ž ti .

g ⬘ Ž zi . y g ⬘ Ž ti . ⌬ x i .

Ž 6.49 .

is1

Since f Ž x . is bounded on w a, b x and g ⬘Ž x . is uniformly continuous on w a, b x by Theorem 3.4.6, then for a given ⑀ ) 0 there exists a ␦ 1 ) 0, which depends only on ⑀ , such that g ⬘ Ž z i . y g ⬘ Ž ti . -

⑀ 2 M Ž b y a.

,

if < z i y t i < - ␦ 1 , where M) 0 is such that < f Ž x .< F M on w a, b x. From Ž6.49. it follows that if the partition P has a norm ⌬ p - ␦ 1 , then S Ž P , f , g . y S Ž P , h. -

⑀ 2

Ž 6.50 .

.

Now, since f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x, then by definition, for the given ⑀ ) 0 there exists a ␦ 2 ) 0 such that SŽ P , f , g . y

Ha f Ž x . dg Ž x . b

-

⑀ 2

,

Ž 6.51 .

if the norm ⌬ p of P is less than ␦ 2 . We conclude from Ž6.50. and Ž6.51. that if the norm of P is less than minŽ ␦ 1 , ␦ 2 ., then S Ž P , h. y

Ha f Ž x . dg Ž x . b

-⑀ .

Since ⑀ is arbitrary, this inequality implies that Hab f Ž x . dg Ž x . is in fact the Riemann integral Hab hŽ x . dxs Hab f Ž x . g ⬘Ž x . dx. I Using Theorem 6.8.3, it is easy to see that if, for example, f Ž x . s 1 and Ž g x . s x 2 , then Hab f Ž x . dg Ž x . s Hab f Ž x . g ⬘Ž x . dxs Hab 2 x dxs b 2 y a2 . It should be noted that Theorems 6.8.1 and 6.8.2 provide sufficient conditions for the existence of Hab f Ž x . dg Ž x .. It is possible, however, for the Riemann᎐Stieltjes integral to exist even if g Ž x . is a discontinuous function. For example, consider the function g Ž x . s ␥ I Ž x y c ., where ␥ is a nonzero

RIEMANN᎐ STIELTJES INTEGRAL

237

constant, a- c - b, and I Ž x y c . is such that I Ž xy c . s

½

0, 1,

x- c, xG c.

The quantity ␥ represents what is called a jump at x s c. If f Ž x . is bounded on w a, b x and is continuous at xs c, then

Ha f Ž x . dg Ž x . s ␥ f Ž c . . b

Ž 6.52 .

To show the validity of formula Ž6.52., let Ps
x 0 , x 1 , . . . , x n4 be any partition of w a, b x. Then, ⌬ g i s g Ž x i . y g Ž x iy1 . will be zero as long as x i - c or x iy1 G c. Suppose, therefore, that there exists a k, 1 F k F n, such that x ky1 - c F x k . In this case, the sum SŽ P, f, g . in formula Ž6.44. takes the form SŽ P , f , g . s

n

Ý f Ž ti . ⌬ g i s ␥ f Ž tk . .

is1

It follows that S Ž P , f , g . y ␥ f Ž c . s < ␥ < f Ž tk . y f Ž c . .

Ž 6.53 .

Now, let ⑀ ) 0 be given. Since f Ž x . is continuous at xs c, then there exists a ␦ ) 0 such that f Ž tk . y f Ž c . -

⑀ <␥<

,

if < t k y c < - ␦ . Thus if the norm ⌬ p of P is chosen so that ⌬ p - ␦ , then SŽ P , f , g . y␥ f Ž c. -⑀ .

Ž 6.54 .

Equality Ž6.52. follows from comparing inequalities Ž6.45. and Ž6.54.. It is now easy to show that if gŽ x. s

½

␭, ␭⬘,

aF x- b, xs b,

and if f Ž x . is continuous at xs b, then

Ha f Ž x . dg Ž x . s Ž ␭⬘ y ␭. f Ž b . . b

Ž 6.55 .

The previous examples represent special cases of a class of functions defined on w a, b x called step functions. These functions are constant on w a, b x

238

INTEGRATION

except for a finite number of jump discontinuities. We can generalize formula Ž6.55. to this class of functions as can be seen in the next theorem. Theorem 6.8.4. Let g Ž x . be a step function defined on w a, b x with jump discontinuities at xs c1 , c 2 , . . . , c n , and a- c1 - c 2 - ⭈⭈⭈ - c n s b, such that

°␭ ,

aF x- c1 , c1 F x- c 2 , . . . c ny1 F x- c n , xs c n .

1

␭2 ,

. g Ž x . s~ ..

␭n ,

¢␭

nq1 ,

If f Ž x . is bounded on w a, b x and is continuous at xs c1 , c 2 , . . . , c n , then n

Ha f Ž x . dg Ž x . s Ý Ž ␭ b

iq1 y ␭ i

. f Ž ci . .

Ž 6.56 .

is1

Proof. The proof can be easily obtained by first writing the integral in formula Ž6.56. as

Ha f Ž x . dg Ž x . sHa

c1

b

f Ž x . dg Ž x . q

Hc

c2

f Ž x . dg Ž x .

1

Hc

cn

q ⭈⭈⭈ q

f Ž x . dg Ž x . .

Ž 6.57 .

ny1

If we now apply formula Ž6.55. to each integral in Ž6.57. we obtain

Ha

c1

f Ž x . dg Ž x . s Ž ␭ 2 y ␭1 . f Ž c1 . ,

Hc

c2

f Ž x . dg Ž x . s Ž ␭ 3 y ␭2 . f Ž c 2 . ,

1

. . .

Hc

cn

f Ž x . dg Ž x . s Ž ␭ nq1 y ␭ n . f Ž c n . .

ny1

By adding up all these integrals we obtain formula Ž6.56..

I

EXAMPLE 6.8.1. One example of a step function is the greatest-integer function w x x, which is defined as the greatest integer less than or equal to x. If f Ž x . is bounded on w0, n x and is continuous at xs 1, 2, . . . , n, where n is a

239

APPLICATIONS IN STATISTICS

positive integer, then by Theorem 6.8.4 we can write n

H0 f Ž x . d w x x s Ý f Ž i . . n

Ž 6.58 .

is1

It follows that every finite sum of the form Ý nis1 a i can be expressed as a Riemann᎐Stieltjes integral with respect to w x x of a function f Ž x . continuous on w0, n x such that f Ž i . s a i , i s 1, 2, . . . , n. The Riemann᎐Stieltjes integral has therefore the distinct advantage of making finite sums expressible as integrals.

6.9. APPLICATIONS IN STATISTICS Riemann integration plays an important role in statistical distribution theory. Perhaps the most prevalent use of the Riemann integral is in the study of the distributions of continuous random variables. We recall from Section 4.5.1 that a continuous random variable X with a cumulative distribution function F Ž x . s P Ž X F x . is absolutely continuous if F Ž x . is differentiable. In this case, there exists a function f Ž x . called the density function of X such that F⬘Ž x . s f Ž x ., that is, FŽ x. s

Hy⬁ f Ž t . dt. x

Ž 6.59 .

In general, if X is a continuous random variable, it need not be absolutely continuous. It is true, however, that most common distributions that are continuous are also absolutely continuous. The probability distribution of an absolutely continuous random variable is completely determined by its density function. For example, from Ž6.59. it follows that P Ž a- X - b . s F Ž b . y F Ž a . s

Ha f Ž x . dx. b

Ž 6.60 .

Note that the value of this probability remains unchanged if one or both of the end points of the interval w a, b x are included. This is true because the probability assigned to these individual points is zero when X has a continuous distribution. The mean ␮ and variance ␴ 2 of X are given by

␮sEŽ X . s

⬁

Hy⬁ xf Ž x . dx,

␴ 2 s Var Ž X . s

⬁

Hy⬁ Ž xy ␮ .

2

f Ž x . dx.

240

INTEGRATION

In general, the kth central moment of X, denoted by ␮ k Ž k s 1, 2, . . . ., is defined as

␮k s E Ž X y ␮ .

k

s

⬁

Hy⬁ Ž xy ␮ .

k

f Ž x . dx.

Ž 6.61 .

We note that ␴ 2 s ␮ 2 . Similarly, the kth noncentral moment of X, denoted by ␮Xk Ž k s 1, 2, . . . ., is defined as

␮Xk s E Ž X k . .

Ž 6.62 .

The first noncentral moment of X is its mean ␮ , while its first central moment is equal to zero. We note that if the domain of the density function f Ž x . is infinite, then ␮ k and ␮Xk are improper integrals of the first kind. Therefore, they may or may ⬁ <
Ž 6.63 .

if j F k,

which is true because < x < j F < x < k if < x < G 1 and < x < j - 1 if < x < - 1. Hence, < x j < F < x < k q 1 for all x. Consequently, from Ž6.63. we obtain the inequality

␯ j F ␯ k q 1,

jFk,

which implies that ␮Xj exists for j F k. Since the central moment ␮ j in formula Ž6.61. is expressible in terms of noncentral moments of order j or smaller, the existence of ␯ j also implies the existence of ␮ j . EXAMPLE 6.9.1.

Consider a random variable X with the density function f Ž x. s

1

␲ Ž1qx2 .

y⬁ - x- ⬁.

,

Such a random variable has the so-called Cauchy distribution. Its mean ␮ does not exist. This follows from the fact that in order of ␮ to exist, the two limits in the following formula must exist:

␮s

1

x dx

H ␲ a™⬁ ya 1 q x lim

0

2

q

1

H ␲ b™⬁ 0 lim

b

x dx 1qx2

.

Ž 6.64 .

241

APPLICATIONS IN STATISTICS

0 But, Hya x dxrŽ1 q x 2 . s y 12 logŽ1 q a2 . ™ y⬁ as a™ ⬁, and H0b x dxrŽ1 q x 2 . 1 ⬁ s 2 logŽ1 q b 2 . ™ ⬁ as b™ ⬁. The integral Hy⬁ x dxrŽ1 q x 2 . is therefore divergent, and hence ␮ does not exist. It should be noted that it would be incorrect to state that

1

x dx

H ␲ a™⬁ ya 1 q x

␮s

a

lim

2

Ž 6.65 .

,

which is equal to zero. This is because the limits in Ž6.64. must exist for any a and b that tend to infinity. The limit in formula Ž6.65. requires that as b. Such a limit is therefore considered as a subsequential limit. The higher-order moments of the Cauchy distribution do not exist either. It is easy to verify that

␮Xk s

1

⬁

x k dx

H ␲ y⬁ 1 q x

Ž 6.66 .

2

is divergent for k G 1. EXAMPLE 6.9.2. Consider a random variable X that has the logistic distribution with the density function f Ž x. s

ex

Ž1qe x .

2

y⬁ - x - ⬁.

,

The mean of X is

␮s s

xe x

⬁

Hy⬁ Ž 1 q e

.

x 2

dx

H0 log ž 1 y u / du, u

1

Ž 6.67 .

where u s e xrŽ1 q e x .. We recognize the integral in Ž6.67. as being an improper integral of the second kind with singularities at u s 0 and u s 1. We therefore write

␮ s limq a™0

Ha

1r2

log

ž

u 1yu

/

duq limy b™1

H1r2 log ž 1 y u / du. b

u

Ž 6.68 .

242

INTEGRATION

Thus,

␮ s limq u log u q Ž 1 y u . log Ž 1 y u . a™0

1r2 a

q limy u log u q Ž 1 y u . log Ž 1 y u . b™1

b 1r2

s limy b log b q Ž 1 y b . log Ž 1 y b . b™1

y limq a log aq Ž 1 y a . log Ž 1 y a . . a™0

Ž 6.69 .

By applying l’Hospital’s rule ŽTheorem 4.2.6. we find that lim Ž 1 y b . log Ž 1 y b . s limq a log as 0.

b™1 y

a™0

We thus have

␮ s limy Ž b log b . y limq Ž 1 y a . log Ž 1 y a . s 0. b™1

a™0

The variance ␴ 2 of X can be shown to be equal to ␲ 2r3 Žsee Exercise 6.24.. 6.9.1. The Existence of the First Negative Moment of a Continuous Distribution Let X be a continuous random variable with a density function f Ž x .. By definition, the first negative moment of X is E Ž Xy1 .. The existence of such a moment will be explored in this section. The need to evaluate a first negative moment can arise in many practical applications. Here are some examples. EXAMPLE 6.9.3. Let P be a population with a mean ␮ and a variance ␴ 2 . The coefficient of variation is a measure of variation in the population per unit mean and is equal to ␴r < ␮ < , assuming that ␮ / 0. An estimate of this ratio is sr < y < , where s and y are, respectively, the sample standard deviation and sample mean of a sample randomly chosen from P. If the population is normally distributed, then y is also normally distributed and is statistically independent of s. In this case, E Ž sr < y < . s E Ž s . E Ž1r < y < .. The question now is whether E Ž1r < y < . exists or not. EXAMPLE 6.9.4 ŽCalibration or Inverse Regression.. linear regression model E Ž y . s ␤ 0 q ␤ 1 x.

Consider the simple

Ž 6.70 .

In most regression situations, the interest is in predicting the response y for

243

APPLICATIONS IN STATISTICS

a given value of x. For this purpose we use the prediction equation ˆ y s ␤ˆ0 q ␤ˆ1 x, where ␤ˆ0 and ␤ˆ1 are the least-squares estimators of ␤ 0 and ␤ 1 , respectively. These are obtained from the data set
Ž x 1 , y 1 ., Ž x 2 , y 2 ., . . . , Ž x n , yn .4 that results from running n experiments in which y is measured for specified settings of x. There are other situations, however, where the interest is in predicting the value of x, say x 0 , that corresponds to an observed value of y, say y 0 . This is an inverse regression problem known as the calibration problem Žsee Graybill, 1976, Section 8.5; Montgomery and Peck, 1982, Section 9.7.. For example, in calibrating a new type of thermometer, n readings, y 1 , y 2 , . . . , yn , are taken at predetermined known temperature values, x 1 , x 2 , . . . , x n Žthese values are known by using a standard temperature gauge.. Suppose that the relationship between the x i ’s and the yi ’s is well represented by the model in Ž6.70.. If a new reading y 0 is observed using the new thermometer, then it is of interest to estimate the correct temperature x 0 Žthat is, the temperature on the standard gauge corresponding to the observed temperature reading y 0 .. In another calibration problem, the date of delivery of a pregnant woman can be estimated by the size y of the head of her unborn child, which can be determined by a special electronic device Žsonogram.. If the relationship between y and the number of days x left until delivery is well represented by model Ž6.70., then for a measured value of y, say y 0 , it is possible to estimate x 0 , the corresponding value of x. In general, from model Ž6.70. we have E Ž y 0 . s ␤ 0 q ␤ 1 x 0 . If ␤ 1 / 0, we can solve for x 0 and obtain x0 s

E Ž y0 . y ␤0

␤1

.

Hence, to estimate x 0 we use

ˆx 0 s

y 0 y ␤ˆ0

␤ˆ1

s xq

y0 y y

␤ˆ1

,

since ␤ˆ0 s y y ␤ˆ1 x, where xs Ž1rn.Ý nis1 x i , y s Ž1rn.Ý nis1 yi . If the response y is normally distributed with a variance ␴ 2 , then y and ␤ˆ1 are statistically independent. Since y 0 is also statistically independent of ␤ˆ1 Ž y 0 does not x 0 is belong to the data set used to estimate ␤ 1 ., then the expected value of ˆ given by y0 y y EŽ ˆ x 0 . s xq E ␤ˆ1

ž /

s xq E Ž y 0 y y . E

ž / 1

␤ˆ1

Here again it is of interest to know if E Ž1r␤ˆ1 . exists.

.

244

INTEGRATION

Now, suppose that the density function f Ž x . of the continuous random variable X is defined on Ž0, ⬁.. Let us also assume that f Ž x . is continuous. The expected value of Xy1 is E Ž Xy1 . s

f Ž x.

⬁

H0

x

Ž 6.71 .

dx.

This is an improper integral with a singularity at x s 0. In particular, if f Ž0. ) 0, then E Ž Xy1 . does not exist, because f Ž x . rx

limq

1rx

x™0

s f Ž 0 . ) 0.

By Theorem 6.5.7, the integrals H0⬁Ž f Ž x .rx . dx and H0⬁Ž dxrx . are of the same kind. Since the latter is divergent, then so is the former. Note that if f Ž x . is defined on Žy⬁, ⬁. and f Ž0. ) 0, then E Ž Xy1 . does not exist either. In this case, ⬁

Hy⬁

f Ž x. x

dxs

Hy⬁ 0

f Ž x. x

Hy⬁

sy

0

dxq

f Ž x. < x<

⬁

H0

dxq

f Ž x. x ⬁

H0

dx

f Ž x. x

dx.

Both integrals on the right-hand side are divergent. A sufficient condition for the existence of E Ž Xy1 . is given by the following theorem wsee Piegorsch and Casella Ž1985.x: Theorem 6.9.1. Let f Ž x . be a continuous density function for a random variable X defined on Ž0, ⬁.. If limq

f Ž x. x␣

x™0

for some ␣ ) 0,

-⬁

Ž 6.72 .

then E Ž Xy1 . exists. Proof. Since the limit of f Ž x .rx ␣ is finite as x™ 0q, there exist finite constants M and ␦ ) 0 such that f Ž x .rx ␣ - M if 0 - x- ␦ . Hence,

H0

␦

f Ž x. x

H0

dx- M

␦ ␣y1

x

dxs

M␦ ␣

␣

.

245

APPLICATIONS IN STATISTICS

Thus, E Ž Xy1 . s -

⬁

H0

f Ž x. x

M␦ ␣

␣

q

dxs 1

⬁

H ␦ ␦

H0

␦

f Ž x. x

dxq

f Ž x . dxF

⬁

H␦

M␦ ␣

␣

f Ž x. x

q

1

␦

dx

- ⬁.

I

It should be noted that the condition of Theorem 6.9.1 is not a necessary one. Piegorsch and Casella Ž1985. give an example of a family of density functions that all violate condition Ž6.72., with some members having a finite first negative moment and others not having one Žsee Exercise 6.25.. Corollary 6.9.1. Let f Ž x . be a continuous density function for a random variable X defined on Ž0, ⬁. such that f Ž0. s 0. If f ⬘Ž0. exists and is finite, then E Ž Xy1 . exists. Proof. We have that f ⬘ Ž 0 . s limq

f Ž x . y f Ž 0. x

x™0

s limq x™0

f Ž x. x

.

By applying Theorem 6.9.1 with ␣ s 1 we conclude that E Ž Xy1 . exists.

I

EXAMPLE 6.9.5. Let X be a normal random variable with a mean ␮ and a variance ␴ 2 . Its density function is given by f Ž x. s

1

'2␲␴ 2

exp y

1 2␴ 2

2 Ž xy ␮ . ,

y⬁ - x - ⬁.

In this example, f Ž0. ) 0. Hence, E Ž Xy1 . does not exist. Consequently, E Ž1r < y < . in Example 6.9.3 does not exist if the population P is normally distributed, since the density function of < y < is positive at zero. Also, in Example 6.9.4, E Ž1r␤ˆ1 . does not exist, because ␤ˆ1 is normally distributed if the response y satisfies the assumption of normality. EXAMPLE 6.9.6. function

Let X be a continuous random variable with the density

f Ž x. s

1 ⌫ Ž nr2. 2 n r2

x n r2y1 eyx r2 ,

0 - x - ⬁,

246

INTEGRATION

where n is a positive integer and ⌫ Ž nr2. is the value of the gamma function, H0⬁eyx x n r2y1 dx. This is the density function of a chi-squared random variable with n degrees of freedom. Let us consider the limit limq

x™0

f Ž x. x

␣

s

1 ⌫ Ž nr2. 2 n r2

lim x n r2y␣y1 eyx r2

x™0 q

for ␣ ) 0. This limit exists and is equal to zero if nr2y ␣ y 1 ) 0, that is, if n ) 2Ž1 q ␣ . ) 2. Thus by Theorem 6.9.1, E Ž Xy1 . exists if the number of degrees of freedom exceeds 2. More recently, Khuri and Casella Ž2002. presented several extensions and generalizations of the results in Piegorsch and Casella Ž1985., including a necessary and sufficient condition for the existence of E Ž Xy1 .. 6.9.2. Transformation of Continuous Random Variables Let Y be a continuous random variable with a density function f Ž y .. Let W s ␺ Ž Y ., where ␺ Ž y . is a function whose derivative exists and is continuous on a set A. Suppose that ␺ ⬘Ž y . / 0 for all y g A, that is, ␺ Ž y . is strictly monotone. We recall from Section 4.5.1 that the density function g Ž w . of W is given by g Ž w . s f ␺y1 Ž w .

d ␺y1 Ž w . dw

,

w g B,

Ž 6.73 .

where B is the image of A under ␺ and y s ␺y1 Ž w . is the inverse function of w s ␺ Ž y .. This result can be easily obtained by applying the change of variables technique in Section 6.4.1. This is done as follows: We have that for any w 1 and w 2 such that w 1 F w 2 , P Ž w1 F W F w 2 . s

Hw

w2

g Ž w . dw.

Ž 6.74 .

1

If ␺ ⬘Ž y . ) 0, then ␺ Ž y . is strictly monotone increasing. Hence, P Ž w1 F W F w 2 . s P Ž y1 F Y F y 2 . ,

Ž 6.75 .

where y 1 and y 2 are such that w 1 s ␺ Ž y 1 ., w 2 s ␺ Ž y 2 .. But P Ž y1 F Y F y 2 . s

Hy

y2

f Ž y . dy.

Ž 6.76 .

1

Let us now apply the change of variables y s ␺y1 Ž w . to the integral in Ž6.76..

247

APPLICATIONS IN STATISTICS

By Theorem 6.4.9 we have

Hy

y2 1

f Ž y . dy s

Hw

w2

y1

f ␺

d ␺y1 Ž w .

Ž w.

dw

1

Ž 6.77 .

dw.

From Ž6.75. and Ž6.76. we obtain P Ž w1 F W F w 2 . s

Hw

w2

d ␺y1 Ž w .

f ␺y1 Ž w .

dw

1

dw.

Ž 6.78 .

On the other hand, if ␺ ⬘Ž y . - 0, then P Ž w 1 F W F w 2 . s P Ž y 2 F Y F y 1 .. Consequently, P Ž w1 F W F w 2 . s s

Hy

y1

f Ž y . dy

2

Hw

w1

f ␺y1 Ž w .

d ␺y1 Ž w .

2

Hw

w2

sy

y1

f ␺

Ž w.

1

dw

dw

d ␺y1 Ž w . dw

dw.

Ž 6.79 .

dw.

Ž 6.80 .

By combining Ž6.78. and Ž6.79. we obtain P Ž w1 F W F w 2 . s

Hw

w2

f ␺y1 Ž w .

1

d ␺y1 Ž w . dw

Formula Ž6.73. now follows from comparing Ž6.74. and Ž6.80.. The Case Where w s ␺ (y) Has No Unique In®erse Formula Ž6.73. requires that w s ␺ Ž y . has a unique inverse, the existence of which is guaranteed by the nonvanishing of the derivative ␺ ⬘Ž y .. Let us now consider the following extension: The function ␺ Ž y . is continuously differentiable, but its derivative can vanish at a finite number of points inside its domain. We assume that the domain of ␺ Ž y . can be partitioned into a finite number, say n, of disjoint subdomains, denoted by I1 , I2 , . . . , In , on each of which ␺ Ž y . is strictly monotone Ždecreasing or increasing .. Hence, on each Ii Ž i s 1, 2, . . . , n., ␺ Ž y . has a unique inverse. Let ␺ i denote the restriction of Ž w ., the function ␺ to Ii , that is, ␺ i Ž y . has a unique inverse, y s ␺y1 i i s 1, 2, . . . , n. Since I1 , I2 , . . . , In are disjoint, for any w 1 and w 2 such that w 1 F w 2 we have P Ž w1 F W F w 2 . s

n

ÝP

is1

Y g ␺y1 i Ž w1 , w 2 . ,

Ž 6.81 .

248

INTEGRATION

Ž w 1 , w 2 . is the inverse image of w w 1 , w 2 x, which is a subset of Ii , where ␺y1 i i s 1, 2, . . . , n. Now, on the ith subdomain we have P Y g ␺y1 i Ž w1 , w 2 . s

H␺

s

HT

y1 Ž w1 , w2 . i

f Ž y . dy

f ␺y1 i Ž w.

d ␺y1 i Ž w.

i s 1, 2, . . . , n,

dw,

dw

i

Ž 6.82 . Ž w 1 , w 2 . under ␺ i . Formula Ž6.82. follows from where Ti is the image of ␺y1 i applying formula Ž6.80. to the function ␺ i Ž y ., i s 1, 2, . . . , n. Note that Ti s Ž w 1 , w 2 .x s w w 1 , w 2 x l ␺ i Ž Ii ., i s 1, 2, . . . , n Žwhy?.. Thus Ti is a subset ␺ i w ␺y1 i of both w w 1 , w 2 x and ␺ i Ž Ii .. We can therefore write the integral in Ž6.82. as

HT f

␺y1 i Ž w.

d ␺y1 i Ž w.

dws

dw

i

w2

d ␺y1 i Ž w.

1

dw

Hw

␦ i Ž w . f ␺y1 i Ž w.

dw, Ž 6.83 .

where ␦ i Ž w . s 1 if w g ␺ i Ž Ii . and ␦ i Ž w . s 0 otherwise, i s 1, 2, . . . , n. Using Ž6.82. and Ž6.83. in formula Ž6.81., we obtain n

ÝH

P Ž w1 F W F w 2 . s

w2

is1

s

w1

␦ i Ž w . f ␺y1 i Ž w.

w2 n

Hw Ý ␦ Ž w . f i

1

␺y1 i Ž w.

d ␺y1 i Ž w. dw d ␺y1 i Ž w.

is1

dw

dw

dw,

from which we deduce that the density function of W is given by g Ž w. s

n

Ý ␦i Ž w . f

␺y1 i Ž w.

is1

EXAMPLE 6.9.7. density function

d ␺y1 i Ž w. dw

.

Ž 6.84 .

Let Y have the standard normal distribution with the

f Ž y. s

1

'2␲

ž /

exp y

y2 2

,

y⬁ - y - ⬁.

Define the random variable W as W s Y 2 . In this case, the function w s y 2

249

APPLICATIONS IN STATISTICS

has two inverse functions on Žy⬁, ⬁., namely, ys

½

y'w ,

'w ,

y F 0, y ) 0.

Thus I1 s Žy⬁, 0x, I2 s Ž0, ⬁., and ␺ 1Ž I1 . s w0, ⬁., ␺ 2 Ž I2 . s Ž0, ⬁.. Hence, ␦ 1Ž w . s 1, ␦ 2 Ž w . s 1 if w g Ž0, ⬁.. By applying formula Ž6.84. we then get g Ž w. s s

1

'2␲

eyw r2

1

eyw r2

'2␲ 'w

y1

1

2'w

q

,

w ) 0.

'2␲

eyw r2

1 2'w

This represents the density function of a chi-squared random variable with one degree of freedom. 6.9.3. The Riemann–Stieltjes Integral Representation of the Expected Value Let X be a random variable with a cumulative distribution function F Ž x .. Suppose that hŽ x . is Riemann᎐Stieltjes integrable with respect to F Ž x . on Žy⬁, ⬁.. Then the expected value of hŽ X . is defined as E hŽ X . s

⬁

Hy⬁h Ž x . dF Ž x . .

Ž 6.85 .

Formula Ž6.85. provides a unified representation of expected values for both discrete and continuous random variables. If X is a continuous random variable with a density function f Ž x ., that is, F⬘Ž x . s f Ž x ., then E hŽ X . s

⬁

Hy⬁h Ž x . f Ž x . dx.

If, however, X has a discrete distribution with a probability mass function pŽ x . and takes the values c1 , c2 , . . . , c n , then E hŽ X . s

n

Ý h Ž ci . p Ž ci . .

Ž 6.86 .

is1

Formula Ž6.86. follows from applying Theorem 6.8.4. Here, F Ž x . is a step

250

INTEGRATION

function with jump discontinuities at c1 , c 2 , . . . , c n such that

°0,

y⬁ - x- c1

p Ž c1 . ,

c1 F x- c 2 ,

p Ž c1 . q p Ž c 2 . , . . .

c 2 F x- c3 ,

~ ny1

FŽ x. s

Ý p Ž ci . ,

c ny1 F x - c n ,

is1 n

¢ Ý p Ž c . s 1,

c n F x - ⬁.

i

is1

Thus, by formula Ž6.56. we obtain n

⬁

Hy⬁h Ž x . dF Ž x . s Ý p Ž c . h Ž c . . i

i

Ž 6.87 .

is1

For example, suppose that X has the discrete uniform distribution with pŽ x . s 1rn for xs c1 , c 2 , . . . , c n . Its cumulative distribution function F Ž c . can be expressed as FŽ x. sPw XFxx s

1 n

n

Ý I Ž xy c i . ,

is1

where I Ž x y c i . is equal to zero if x- c i and is equal to one if xG c i Ž i s 1, 2, . . . , n.. The expected value of hŽ X . is E hŽ X . s

1 n

n

Ý h Ž ci . .

is1

EXAMPLE 6.9.8. The moment generating function ␾ Ž t . of a random variable X with a cumulative distribution function F Ž x . is defined as the expected value of h t Ž X . s e t X , that is,

␾ Ž t . sEŽ et X . s

⬁

Hy⬁e

tx

dF Ž x . ,

where t is a scalar. If X is a discrete random variable with a probability mass function pŽ x . and takes the values c1 , c 2 , . . . , c n , . . . , then by letting n go to infinity in Ž6.87. we obtain Žsee also Section 5.6.2.

␾Ž t. s

⬁

Ý p Ž ci . e t c . i

is1

251

APPLICATIONS IN STATISTICS

The moment generating function of a continuous random variable with a density function f Ž x . is

␾Ž t. s

⬁

Hy⬁e

tx

f Ž x . dx.

Ž 6.88 .

The convergence of the integral in Ž6.88. depends on the choice of the scalar t. For example, for the gamma distribution G Ž ␣ , ␤ . with the density function f Ž x. s

x ␣y1 eyx r ␤ ⌫Ž ␣ . ␤ ␣

␣ ) 0,

,

␤ ) 0, 0 - x- ⬁,

␾ Ž t . is of the form ⬁

␾Ž t. s

H0

s

H0

⬁

e t x x ␣y1 eyx r ␤

dx

⌫Ž ␣ . ␤ ␣

x ␣y1 exp yx Ž 1 y ␤ t . r␤ ⌫Ž ␣ . ␤ ␣

dx.

If we set y s x Ž1 y ␤ t .r␤ , we obtain

␾Ž t. s

H0

ž

␤

⬁

␤y

Ž1y␤ t . ⌫Ž ␣ . ␤ ␣ 1y␤ t

/

␣y1

eyy dy.

Thus,

␾Ž t. s

1

⬁

Ž1y␤ t .

s Ž1y␤ t .

␣

H0

y␣

y ␣y1 eyy ⌫Ž ␣ .

dy

,

since H0⬁ eyy y ␣y1 dy s ⌫ Ž ␣ . by the definition of the gamma function. We note that ␾ Ž t . exists for all values of ␣ provided that 1 y ␤ t ) 0, that is, t - 1r␤ . 6.9.4. Chebyshev’s Inequality In Section 5.6.1 there was a mention of Chebyshev’s inequality. Using the Riemann᎐Stieltjes integral representation of the expected value, it is now possible to provide a proof for this important inequality. Theorem 6.9.2. Let X be a random variable Ždiscrete or continuous. with a mean ␮ and a variance ␴ 2 . Then, for any positive constant r, P Ž < X y ␮ < G r␴ . F

1 r2

.

252

INTEGRATION

Proof. By definition, ␴ 2 is the expected value of hŽ X . s Ž X y ␮ . 2 . Thus,

␴ 2s

⬁

Hy⬁ Ž xy ␮ .

2

dF Ž x . ,

Ž 6.89 .

where F Ž x . is the cumulative distribution function of X. Let us now partition Žy⬁, ⬁. into three disjoint intervals: Žy⬁, ␮ y r␴ x, Ž ␮ y r␴ , ␮ q r␴ ., w ␮ q r␴ , ⬁.. The integral in Ž6.89. can therefore be written as

␴ 2s

␮yr ␴

Hy⬁

2 Ž xy ␮ . dF Ž x . q H

␮yr ␴

⬁

H␮qr ␴ Ž xy ␮ .

q G

␮yr ␴

Hy⬁

␮qr ␴

2

2 Ž xy ␮ . dF Ž x .

dF Ž x . ⬁

2 Ž xy ␮ . dF Ž x . q H

␮qr ␴

2 Ž xy ␮ . dF Ž x . .

Ž 6.90 .

We note that in the first integral in Ž6.90., xF ␮ y r␴ , so that x y ␮ F yr␴ . Hence, Ž xy ␮ . 2 G r 2␴ 2 . Also, in the second integral, xy ␮ G r␴ . Hence, Ž xy ␮ . 2 G r 2␴ 2 . Consequently, ␮yr ␴

Hy⬁

2 Ž xy ␮ . dF Ž x . G r 2␴ 2H

␮yr ␴

y⬁

⬁

H␮qr ␴ Ž xy ␮ .

2

dF Ž x . G r 2␴ 2

dF Ž x . s r 2␴ 2 P Ž X F ␮ y r␴ . ,

⬁

H␮qr ␴dF Ž x . s r ␴ 2

2

P Ž X G ␮ q r␴ . .

From inequality Ž6.90. we then have

␴ 2 G r 2␴ 2 P Ž X y ␮ F yr␴ . q P Ž X y ␮ G r␴ . s r 2␴ 2 P Ž < X y ␮ < G r␴ . , which implies that P Ž < X y ␮ < G r␴ . F

1 r2

.

I

FURTHER READING AND ANNOTATED BIBLIOGRAPHY DeCani, J. S., and R. A. Stine Ž1986.. ‘‘A note on deriving the information matrix for a logistic distribution.’’ Amer. Statist., 40, 220᎐222. ŽThis article uses calculus techniques, such as integration and l’Hospital’s rule, in determining the mean and variance of the logistic distribution as was seen in Example 6.9.2.. Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. ŽChap. 5 discusses the Riemann integral; Chap. 16 provides a study of improper integrals. .

253

EXERCISES

Graybill, F. A. Ž1976.. Theory and Application of the Linear Model. Duxbury Press, North Scituate, Massachusetts. ŽSection 8.5 discusses the calibration problem for a simple linear regression model as was seen in Example 6.9.4.. Hardy, G. H., J. E. Littlewood, and G. Polya ´ Ž1952.. Inequalities, 2nd ed. Cambridge University Press, Cambridge, England. ŽThis is a classic and often referenced book on inequalities. Chap. 6 is relevant to the present chapter. . Hartig, D. Ž1991.. ‘‘L’Hopital’s rule via integration.’’ Amer. Math. Monthly, 98, ˆ 156᎐157. Khuri, A. I., and G. Casella Ž2002.. ‘‘The existence of the first negative moment revisited.’’ Amer. Statist., 56, 44᎐47. ŽThis article demonstrates the utility of the comparison test given in Theorem 6.5.3 in showing the existence of the first negative moment of a continuous random variable.. Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽSection 2.2.2 gives the Riemann᎐Stieltjes integral representation of the expected value of a random variable as was seen in Section 6.9.3.. Montgomery, D. C., and E. A. Peck Ž1982.. Introduction to Linear Regression Analysis. Wiley, New York. ŽThe calibration problem for a simple linear regression model is discussed in Section 9.7.. Moran, P. A. P. Ž1968.. An Introduction to Probability Theory. Clarendon Press, Oxford, England. ŽSection 5.9 defines moments of a random variable using the Riemann᎐Stieltjes integral representation of the expected value; Section 5.10 discusses a number of inequalities pertaining to these moments.. Piegorsch, W. W., and G. Casella Ž1985.. ‘‘The existence of the first negative moment.’’ Amer. Statist., 39, 60᎐62. ŽThis article gives a sufficient condition for the existence of the first negative moment of a continuous random variable as was seen in Section 6.9.1.. Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts. ŽChap. 9 is concerned with transformations of continuous random variables as was seen in Section 6.9.2. See, in particular, Theorems 2 and 3 in this chapter. . Taylor, A. E., and W. R. Mann Ž1972.. Ad®anced Calculus, 2nd ed. Wiley, New York. ŽChap. 18 discusses the Riemann integral as well as the Riemann᎐Stieltjes integral; improper integrals are studied in Chap. 22.. Wilks, S. S. Ž1962.. Mathematical Statistics. Wiley, New York. ŽChap. 3 uses the Riemann᎐Stieltjes integral to define expected values and moments of random variables; functions of random variables are discussed in Section 2.8..

EXERCISES In Mathematics 6.1. Let f Ž x . be a bounded function defined on the interval w a, b x. Let P be a partition of w a, b x. Show that f Ž x . is Riemann integrable on w a, b x if and only if inf US P Ž f . s sup LSP Ž f . s P

P

Ha f Ž x . dx. b

254

INTEGRATION

6.2. Construct a function that has a countable number of discontinuities in w0, 1x and is Riemann integrable on w0, 1x. 6.3. Show that if f Ž x . is continuous on w a, b x except for a finite number of discontinuities of the first kind Žsee Definition 3.4.2., then f Ž x . is Riemann integrable on w a, b x. 6.4. Show that the function f Ž x. s

½

x cos Ž ␲r2 x . , 0,

0 - xF 1, xs 0,

is not of bounded variation on w0, 1x. 6.5. Let f Ž x . and g Ž x . have continuous derivatives with g ⬘Ž x . ) 0. Suppose that lim x ™⬁ f Ž x . s ⬁, lim x ™⬁ g Ž x . s ⬁, and lim x ™⬁ f ⬘Ž x .rg ⬘Ž x . s L, where L is finite. (a) Show that for a given ⑀ ) 0 there exists a constant M) 0 such that for x) M, f ⬘ Ž x . y Lg⬘ Ž x . - ⑀ g ⬘ Ž x . . Hence, if ␭1 and ␭2 are such that M- ␭1 - ␭ 2 , then

H␭

␭2

f ⬘ Ž x . y Lg⬘ Ž x . dx -

1

H␭

␭2

⑀ g ⬘ Ž x . dx.

1

(b) Deduce from Ža. that f Ž ␭2 . g Ž ␭2 .

yL -⑀q

f Ž ␭1 . g Ž ␭2 .

q < L<

g Ž ␭1 . g Ž ␭2 .

.

(c) Make use of Žb. to show that for a sufficiently large ␭2 , f Ž ␭2 . g Ž ␭2 .

y L - 3⑀ ,

and hence lim x ™⬁ f Ž x .rg Ž x . s L. w Note: This problem verifies l’Hospital’s rule for the ⬁r⬁ indeterminate form by using integration properties without relying on Cauchy’s mean value theorem as in Section 4.2 Žsee Hartig, 1991.x. 6.6. Show that if f Ž x . is continuous on w a, b x, and if g Ž x . is a nonnegative Riemann integrable function on w a, b x such that Hab g Ž x . dx) 0, then

255

EXERCISES

there exists a constant c, aF c F b, such that Hab f Ž x . g Ž x . dx Hab g Ž x . dx

sf Ž c. .

6.7. Prove Theorem 6.5.2. 6.8. Prove Theorem 6.5.3. 6.9. Suppose that f Ž x . is a positive monotone decreasing function defined on w a, ⬁. such that f Ž x . ™ 0 as x ™ ⬁. Show that if f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x for every bG a, where g Ž x . is bounded on w a, ⬁., then the integral Ha⬁ f Ž x . dg Ž x . is convergent. 6.10. Show that lim n™⬁ H0n␲ < Žsin x .rx < dxs ⬁, where n is a positive integer. w Hint: Show first that n␲

H0

sin x x

dxs

␲

H0 sin x

1 x

q

1 xq␲

q ⭈⭈⭈ q

1 xq Ž n y 1 . ␲

dx. x

6.11. Apply Maclaurin’s integral test to determine convergence or divergence of the following series: ⬁

log n

( a)

Ý ' ns1 n n

( b)

Ý

⬁

nq4

ns1

2 n3 q 1

⬁

( c)

,

,

1

. Ý ' ns1 n q 1 y 1

6.12. Consider the sequence
f nŽ x .4⬁ns1 , where f nŽ x . s nxrŽ1 q nx 2 ., x G 0. Find the limit of H12 f nŽ x . dx as n ™ ⬁. 6.13. Consider the improper integral H01 x my1 Ž1 y x . ny1 dx.

256

INTEGRATION

(a) Show that the integral converges if m ) 0, n ) 0. In this case, the function B Ž m, n. defined as B Ž m, n . s

H0 x

1 my1

Ž1yx.

ny1

dx

is called the beta function. (b) Show that

H0

B Ž m, n . s 2

␲r2

sin 2 my1 ␪ cos 2 ny1 ␪ d␪

(c) Show that B Ž m, n . s

x my1

⬁

H0

Ž1qx.

mqn

dxs

x ny1

⬁

H0

Ž1qx.

mqn

dx

(d) Show that B Ž m, n . s

H0

1

x my1 q x ny1

Ž1qx.

mqn

dx.

6.14. Determine whether each of the following integrals is convergent or divergent:

( a)

⬁

dx

H0 '1 q x ⬁

( b)

H0

( c)

H0

( d)

H0

⬁

,

dx

Ž1qx3. 1

3

1r3

,

1r3

,

dx

Ž1yx3. dx

'x Ž 1 q 2 x .

.

6.15. Let f 1Ž x . and f 2 Ž x . be bounded on w a, b x, and g Ž x . be monotone increasing on w a, b x. If f 1Ž x . and f 2 Ž x . are Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x, then show that f 1Ž x . f 2 Ž x . is also Riemann᎐Stieltjes integrable with respect to g Ž x . on w a, b x.

257

EXERCISES

6.16. Let f Ž x . be a function whose first n derivatives are continuous on w a, b x, and let

Ž by x . f Ž ny1. Ž x . . Ž n y 1. ! ny1

h n Ž x . s f Ž b . y f Ž x . y Ž by x . f ⬘ Ž x . y ⭈⭈⭈ y Show that h n Ž a. s

1

H Ž by x . Ž n y 1. ! a b

ny1

f Ž n. Ž x . dx

and hence

Ž by a. f Ž b . s f Ž a . q Ž by a . f ⬘ Ž a . q ⭈⭈⭈ q f Ž ny1. Ž a . Ž n y 1. ! ny1

q

1

Ha Ž by x . b

Ž n y 1. !

ny1

f Ž n. Ž x . dx.

This represents Taylor’s expansion of f Ž x . around xs a Žsee Section 4.3. with a remainder R n given by Rn s

1

H Ž by x . Ž n y 1. ! a b

ny1

f Ž n. Ž x . dx.

w Note: This form of Taylor’s theorem has the advantage of providing an exact formula for R n , which does not involve an undetermined number ␪n as was seen in Section 4.3.x 6.17. Suppose that f Ž x . is monotone and its derivative f ⬘Ž x . is Riemann integrable on w a, b x. Let g Ž x . be continuous on w a, b x. Show that there exists a number c, aF c F b, such that

Ha f Ž x . g Ž x . dxs f Ž a. Ha g Ž x . dxq f Ž b . Hc g Ž x . dx. c

b

b

6.18. Deduce from Exercise 6.17 that for any b) a) 0,

Ha

b

6.19. Prove Lemma 6.7.1.

sin x x

dx F

4 a

.

258

INTEGRATION

6.20. Show that if f 1Ž x ., f 2 Ž x ., . . . , f nŽ x . are such that < f i Ž x .< p is Riemann integrable on w a, b x for i s 1, 2, . . . , n, where 1 F p- ⬁, then n

Ha Ý f Ž x . b

i

is1

1rp

p

dx

n

F

Ha

b

Ý

is1

fi Ž x .

p

1rp

dx

.

6.21. Prove Theorem 6.8.1. 6.22. Prove Theorem 6.8.2. In Statistics 6.23. Show that the integral in formula Ž6.66. is divergent for k G 1. 6.24. Consider the random variable X that has the logistic distribution described in Example 6.9.2. Show that VarŽ X . s ␲ 2r3. 6.25. Let
f nŽ x .4⬁ns1 be a family density functions defined by

fn Ž x . s

< log n x < y1 H0␭ < log n t < y1 dt

,

0 - x- ␭ ,

where ␭ g Ž0, 1.. (a) Show that condition Ž6.72. of Theorem 6.9.1 is not satisfied by any f nŽ x ., n G 1. (b) Show that when n s 1, E Ž Xy1 . does not exist, where X is a random variable with the density function f 1Ž x .. (c) Show that for n ) 1, E Ž Xy1 . exists, where X n is a random variable n Ž with the density function f n x .. 6.26. Let X be a random variable with a continuous density function f Ž x . on Ž0, ⬁.. Suppose that f Ž x . is bounded near zero. Then E Ž Xy␣ . exits, where ␣ g Ž0, 1.. 6.27. Let X be a random variable with a continuous density function f Ž x . on Ž0, ⬁.. If lim x ™ 0q Ž f Ž x .rx ␣ . is equal to a positive constant k for some ␣ ) 0, then E w XyŽ1q␣ . x does not exist.

259

EXERCISES

6.28. The random variable Y has the t-distributions with n degrees of freedom. Its density function is given by ⌫ f Ž y. s

ž

nq1

'n␲ ⌫

/

ž /ž

2

n

1q

y2 n

2

/

y Ž nq1 .r2

y⬁ - y - ⬁,

,

where ⌫ Ž m. is the gamma function H0⬁eyx x my1 dx, m ) 0. Find the density function of W s < Y < . 6.29. Let X be a random variable with a mean ␮ and a variance ␴ 2 . (a) Show that Chebyshev’s inequality can be expressed as P Ž < Xy␮< Gr . F

␴2 r2

,

where r is any positive constant. (b) Let
X n4⬁ns1 be a sequence of independent and identically distributed random variables. If the common mean and variance of the X i ’s are ␮ and ␴ 2 , respectively, then show that P Ž < Xn y ␮ < G r . F

␴2 nr 2

,

where X n s Ž1rn.Ý nis1 X i and r is any positive constant. (c) Deduce from Žb. that X n converges in probability to ␮ as n ™ ⬁, that is, for every ⑀ ) 0, P Ž < Xn y ␮ < G ⑀ . ™ 0

as n ™ ⬁.

6.30. Let X be a random variable with a cumulative distribution function F Ž x .. Let ␮Xk be its kth noncentral moment,

␮Xk s E Ž X k . s

⬁

Hy⬁ x

dF Ž x . .

k

Let ␯ k be the kth absolute moment of X,

␯k sEŽ < X< k . s

⬁

Hy⬁ < x <

Suppose that ␯ k exists for k s 1, 2, . . . , n.

k

dF Ž x . .

260

INTEGRATION

(a) Show that ␯ k2 F ␯ ky1 ␯ kq1 , k s 1, 2, . . . , n y 1. w Hint: For any u and ®, 0F

⬁

Hy⬁

u < x < Ž ky1. r2 q ® < x < Ž kq1. r2

2

dF Ž x .

s u 2␯ ky1 q 2 u®␯ k q ® 2␯ kq1 . x (b) Deduce from Ža. that

␯ 1 F ␯ 21r2 F ␯ 31r3 F ⭈⭈⭈ F ␯n1r n .

CHAPTER 7

Multidimensional Calculus

In the previous chapters we have mainly dealt with real-valued functions of a single variable x. In this chapter we extend the notions of limits, continuity, differentiation, and integration to multivariable functions, that is, functions of several variables. These functions can be real-valued or possibly vector-valued. More specifically, if R n denotes the n-dimensional Euclidean space, n G 1, then we shall in general consider functions defined on a set D ; R n and have values in R m , m G 1. Such functions are represented symbolically as f: D ™ R m , where for x s Ž x 1 , x 2 , . . . , x n .⬘ g D, f Ž x. s f 1Ž x. , f 2 Ž x. , . . . , fm Ž x. ⬘ and f i Žx. is a real-valued function of x 1 , x 2 , . . . , x n Ž i s 1, 2, . . . , m.. Even though the basic framework of the methodology in this chapter is general and applies in any number of dimensions, most of the examples are associated with two- or three-dimensional spaces. At this stage, it would be helpful to review the basic concepts given in Chapters 1 and 2. This can facilitate the understanding of the methodology and its development in a multidimensional environment.

7.1. SOME BASIC DEFINITIONS Some of the concepts described in Chapter 1 pertained to one-dimensional Euclidean spaces. In this section we extend these concepts to higher-dimensional Euclidean spaces. Any point x in R n can be represented as a column vector of the form Ž x 1 , x 2 , . . . , x n .⬘, where x i is the ith element of x Ž i s 1, 2, . . . , n.. The Euclidean norm of x was defined in Chapter 2 Žsee Definition 2.1.4. as 5x 5 2 s ŽÝ nis1 x i2 .1r2 . For simplicity we shall drop the subindex 2 and denote this norm by 5x 5. 261

262

MULTIDIMENSIONAL CALCULUS

Let x 0 g R n. A neighborhood Nr Žx 0 . of x 0 is a set of points in R n that lie within some distance, say r, from x 0 , that is, Nr Ž x 0 . s
x g R n < 5x y x 0 5 - r 4 . If x 0 is deleted from Nr Žx 0 ., we obtain the so-called deleted neighborhood of x 0 , which we denote by Nrd Žx 0 .. A point x 0 in R n is a limit point of a set A ; R n if every neighborhood of x 0 contains an element x of A such that x / x 0 , that is, every deleted neighborhood of x 0 contains points of A. A set A ; R n is closed if every limit point of A belongs to A. A point x 0 in R n is an interior of a set A ; R n if there exists an r ) 0 such that Nr Žx 0 . ; A. A set A ; R n is open if for every point x in A there exists a neighborhood Nr Žx. that is contained in A. Thus A is open if it consists entirely of interior points. A point pg R n is a boundary point of a set A ; R n if every neighborhood of p contains points of A as well as points of A, the complement of A with respect to R n. The set of all boundary points of A is called its boundary and is denoted by Br Ž A.. A set A ; R n is bounded if there exists an r ) 0 such that 5x 5 F r for all x in A. Let g: Jq™ R n be a vector-valued function defined on the set of all positive integers. Let gŽ i . s a i , i G 1. Then
a i 4⬁is1 represents a sequence of points in R n. By a subsequence of
a i 4⬁is1 we mean a sequence
a k i 4⬁is1 such that k 1 - k 2 - ⭈⭈⭈ - k i - ⭈⭈⭈ and k i G i for i G 1 Žsee Definition 5.1.1.. A sequence
a i 4⬁is1 converges to a point c g R n if for a given ⑀ ) 0 there exists an integer N such that 5 a i y c 5 - ⑀ whenever i ) N. This is written symbolically as lim i™⬁ a i s c, or a i ™ c as i ™ ⬁. A sequence
a i 4⬁is1 is bounded if there exists a number K ) 0 such that 5 a i 5 F K for all i. 7.2. LIMITS OF A MULTIVARIABLE FUNCTION We recall from Chapter 3 that for a function of a single variable x, its limit at a point is considered when x approaches the point from two directions, left and right. Here, for a function of several variables, say x 1 , x 2 , . . . , x n , its limit at a point a s Ž a1 , a2 , . . . , a n .⬘ is considered when x s Ž x 1 , x 2 , . . . , x n .⬘ approaches a in any possible way. Thus when n ) 1 there are infinitely many ways in which x can approach a. Definition 7.2.1. Let f: D ™ R m , where D ; R n. Then f Žx. is said to have a limit L s Ž L1 , L2 , . . . , L m .⬘ as x approaches a, written symbolically as x ™ a, where a is a limit point of D, if for a given ⑀ ) 0 there exists a ␦ ) 0 such

263

LIMITS OF A MULTIVARIABLE FUNCTION

that 5f Žx. y L 5 - ⑀ for all x in D l N␦d Ža., where N␦d Ža. is a deleted neighborhood of a of radius ␦ . If it exists, this limit is written symbolically as lim x ™ a f Žx. s L. I Note that whenever a limit of f Žx. exists, its value must be the same no matter how x approaches a. It is important here to understand the meaning of ‘‘x approaches a.’’ By this we do not necessarily mean that x moves along a straight line leading into a. Rather, we mean that x moves closer and closer to a along any curve that goes through a. EXAMPLE 7.2.1.

Consider the behavior of the function

f Ž x1 , x 2 . s

x 13 y x 23 x 12 q x 22

as x s Ž x 1 , x 2 .⬘ ™ 0, where 0 s Ž0, 0.⬘. This function is defined everywhere in R 2 except at 0. It is convenient here to represent the point x using polar coordinates, r and ␪ , such that x 1 s r cos ␪ , x 2 s r sin ␪ , r ) 0, 0 F ␪ F 2␲ . We then have f Ž x1 , x 2 . s

r 3 cos 3 ␪ y r 3 sin 3 ␪ r 2 cos 2 ␪ q r 2 sin 2 ␪

s r Ž cos 3 ␪ y sin 3 ␪ . . Since x ™ 0 if and only if r ™ 0, lim x ™ 0 f Ž x 1 , x 2 . s 0 no matter how x approaches 0. EXAMPLE 7.2.2.

Consider the function f Ž x1 , x 2 . s

x1 x 2 2 x 1 q x 22

.

Using polar coordinates again, we obtain f Ž x 1 , x 2 . s cos ␪ sin ␪ , which depends on ␪ , but not on r. Since ␪ can have infinitely many values, f Ž x 1 , x 2 . cannot be made close to any one constant L no matter how small r is. Thus the limit of this function does not exist as x ™ 0.

264

MULTIDIMENSIONAL CALCULUS

EXAMPLE 7.2.3.

Let f Ž x 1 , x 2 . be defined as f Ž x1 , x 2 . s

x 2 Ž x 12 q x 22 . x 22 q Ž x 12 q x 22 .

2

.

This function is defined everywhere in R 2 except at Ž0, 0.⬘. On the line x 1 s 0, f Ž0, x 2 . s x 23rŽ x 22 q x 24 ., which goes to zero as x 2 ™ 0. When x 2 s 0, f Ž x 1 , 0. s 0 for x 1 / 0; hence, f Ž x 1 , 0. ™ 0 as x 1 ™ 0. Furthermore, for any other straight line x 2 s tx 1 Ž t / 0. through the origin we have f Ž x 1 , tx 1 . s

s

tx 1 Ž x 12 q t 2 x 12 . t 2 x 12 q Ž x 12 q t 2 x 12 . tx 1 Ž 1 q t 2 . t 2 q x 12 Ž 1 q t 2 .

2

2

,

,

x 1 / 0,

which has a limit equal to zero as x 1 ™ 0. We conclude that the limit of f Ž x 1 , x 2 . is zero as x ™ 0 along any straight line through the origin. However, f Ž x 1 , x 2 . does not have a limit as x ™ 0. For example, along the circle x 2 s x 12 q x 22 that passes through the origin, 1 Ž x 12 q x 22 . s , 2 2 Ž x 12 q x 22 . q Ž x 12 q x 22 . 2 2

f Ž x1 , x 2 . s

x 12 q x 22 / 0.

Hence, f Ž x 1 , x 2 . ™ 12 / 0. This example demonstrates that a function may not have a limit as x ™ a even though its limit exists for approaches toward a along straight lines.

7.3. CONTINUITY OF A MULTIVARIABLE FUNCTION The notion of continuity for a function of several variables is much the same as that for a function of a single variable. Definition 7.3.1. Let f: D ™ R m , where D ; R n, and let a g D. Then f Žx. is continuous at a if lim f Ž x . s f Ž a . ,

x™a

where x remains in D as it approaches a. This is equivalent to stating that for a given ⑀ ) 0 there exits a ␦ ) 0 such that 5f Ž x . y f Ž a . 5 - ⑀ for all x g D l N␦ Ža..

265

CONTINUITY OF A MULTIVARIABLE FUNCTION

If f Žx. is continuous at every point x in D, then it is said to be continuous in D. In particular, if f Žx. is continuous in D and if ␦ Žin the definition of continuity. depends only on ⑀ Žthat is, ␦ is the same for all points in D for I the given ⑀ ., then f Žx. is said to be uniformly continuous in D. We now present several theorems that provide some important properties of multivariable continuous functions. These theorems are analogous to those given in Chapter 3. Let us first consider the following lemmas Žthe proofs are left to the reader.: Lemma 7.3.1. quence.

Every bounded sequence in R n has a convergent subse-

This lemma is analogous to Theorem 5.1.4. Lemma 7.3.2. Suppose that f, g: D ™ R are real-valued continuous functions, where D ; R n. Then we have the following: 1. 2. 3.

f q g, f y g, and fg are continuous in D. < f < is continuous in D. 1rf is continuous in D provided that f Žx. / 0 for all x in D.

This lemma is analogous to Theorem 3.4.1. Lemma 7.3.3. Suppose that f: D ™ R m is continuous, where D ; R n, and that g: G ™ R ␯ is also continuous, where G ; R m is the image of D under f. Then the composite function g(f: D ™ R ␯ , defined as g(f Žx. s gwf Žx.x, is continuous in D. This lemma is analogous to Theorem 3.4.2. Theorem 7.3.1. Let f : D ™ R be a real-valued continuous function defined on a closed and bounded set D ; R n. Then there exist points p and q in D for which f Ž p . s sup f Ž x . ,

Ž 7.1 .

xgD

f Ž q . s inf f Ž x . . xgD

Ž 7.2 .

Thus f Žx. attains each of its infimum and supremum at least once in D. Proof. Let us first show that f Žx. is bounded in D. We shall prove this by contradiction. Suppose that f Žx. is not bounded in D. Then we can find a sequence of points
p i 4⬁is1 in D such that < f Žp i .< G i for i G 1 and hence

266

MULTIDIMENSIONAL CALCULUS

< f Žp i .< ™ ⬁ as i ™ ⬁. Since the terms of this sequence are elements in a bounded set,
p i 4⬁is1 must be a bounded sequence. By Lemma 7.3.1, this sequence has a convergent subsequence
p k i 4⬁is1 . Let p 0 be the limit of this subsequence, which is also a limit point of D; hence, it belongs to D, since D is closed. Now, on one hand, < f Žp k i .< ™ < f Žp 0 .< as i ™ ⬁, by the continuity of f Žx. and hence of < f Žx.< wsee Lemma 7.3.2Ž2.x. On the other hand, < f Žp k i .< ™ ⬁. This contradiction shows that f Žx. must be bounded in D. Consequently, the infimum and supremum of f Žx. in D are finite. Suppose now equality Ž7.1. does not hold for any p g D. Then, My f Žx. ) 0 for all x g D, where Ms sup x g D f Žx.. Consequently, w My f Žx.xy1 is positive and continuous in D by Lemma 7.3.2Ž3. and is therefore bounded by the first half of this proof. However, if ␦ ) 0 is any given positive number, then, by the definition of M, we can find a point x ␦ in D for which f Žx ␦ . ) My ␦ , or 1 My f Ž x ␦ .

)

1

␦

.

This implies that w My f Žx.xy1 is not bounded, a contradiction, which proves equality Ž7.1.. The proof of equality Ž7.2. is similar. I Theorem 7.3.2. Suppose that D is a closed and bounded set in R n. If f: D ™ R m is continuous, then it is uniformly continuous in D. Proof. We shall prove this theorem by contradiction. Suppose that f is not uniformly continuous in D. Then there exists an ⑀ ) 0 such that for every ␦ ) 0 we can find a and b in D such that 5 a y b 5 - ␦ , but 5f Ža. y f Žb.5 G ⑀ . Let us choose ␦ s 1ri, i G 1. We can therefore find two sequences
a i 4⬁is1 ,
b i 4⬁is1 with a i , b i g D such that 5 a i y b i 5 - 1ri, and 5f Ž a i . y f Ž b i . 5 G ⑀

Ž 7.3 .

for i G 1. Now, the sequence
a i 4⬁is1 is bounded. Hence, by Lemma 7.3.1, it has a convergent subsequence
a k i 4⬁is1 whose limit, denoted by a 0 , is in D, since D is closed. Also, since f is continuous at a 0 , we can find a ␭ ) 0 such that 5f Žx. y f Ža 0 .5 - ⑀r2 if 5x y a 0 5 - ␭, where x g D. By the convergence of
a k 4⬁is1 to a 0 , we can choose k i large enough so that i 1 ki

-

␭

Ž 7.4 .

2

and 5 ak ya0 5i

␭ 2

.

Ž 7.5 .

267

DERIVATIVES OF A MULTIVARIABLE FUNCTION

From Ž7.5. it follows that 5f Ž a k . y f Ž a 0 . 5 i

⑀ 2

.

Ž 7.6 .

Furthermore, since 5 a k i y b k i 5 - 1rk i , we can write 5b k y a 0 5 F 5 a k y a 0 5 q 5 a k y b k 5 i i i i -

␭ 2

q

1 ki

- ␭.

Hence, by the continuity of f at a 0 , 5 f Žb k . y fŽ a 0 . 5 i

⑀ 2

.

Ž 7.7 .

From inequalities Ž7.6. and Ž7.7. we conclude that whenever k i satisfies inequalities Ž7.4. and Ž7.5., 5 f Ž a k . y f Žb k . 5 F 5 f Ž a k . y f Ž a 0 . 5 q 5 f Žb k . y f Ž a 0 . 5 - ⑀ , i i i i which contradicts inequality Ž7.3.. This leads us to assert that f is uniformly continuous in D. I

7.4. DERIVATIVES OF A MULTIVARIABLE FUNCTION In this section we generalize the concept of differentiation given in Chapter 4 to a multivariable function f: D ™ R m , where D ; R n. Let a s Ž a1 , a2 , . . . , a n .⬘ be an interior point of D. Suppose that the limit

lim

h i™0

f Ž a1 , a2 , . . . , a i q h i , . . . , a n . y f Ž a1 , a2 , . . . , a i , . . . , a n . hi

exists; then f is said to have a partial derivative with respect to x i at a. This derivative is denoted by ⭸ f Ža.r⭸ x i , or just f x iŽa., i s 1, 2, . . . , n. Hence, partial differentiation with respect to x i is done in the usual fashion while treating all the remaining variables as constants. For example, if f : R 3 ™ R is defined

268

MULTIDIMENSIONAL CALCULUS

as f Ž x 1 , x 2 , x 3 . s x 1 x 22 q x 2 x 33 , then at any point x g R 3 we have

⭸ f Ž x. ⭸ x1 ⭸ f Ž x. ⭸ x2 ⭸ f Ž x. ⭸ x3

s x 22 , s 2 x 1 x 2 q x 33 , s 3 x 2 x 32 .

In general, if f j is the jth element of f Ž j s 1, 2, . . . , m., then the terms ⭸ f j Žx.r⭸ x i , for i s 1, 2, . . . , n; j s 1, 2, . . . , m, constitute an m = n matrix called the Jacobian matrix Žafter Carl Gustav Jacobi, 1804᎐1851. of f at x and is denoted by Jf Žx.. If m s n, the determinant of Jf Žx. is called the Jacobian determinant; it is sometimes represented as det Jf Ž x . s

⭸ Ž f1 , f 2 , . . . , fn . ⭸ Ž x1 , x 2 , . . . , x n .

Ž 7.8 .

.

For example, if f: R 3 ™ R 2 is such that f Ž x 1 , x 2 , x 3 . s Ž x 12 cos x 2 , x 22 q x 32 e x 1 . ⬘, then Jf Ž x 1 , x 2 , x 3 . s

2 x 1 cos x 2 x 32

e

x1

yx 12 sin x 2

0

2 x2

2 x 3 e x1

.

Higher-order partial derivatives of f are defined similarly. For example, the second-order partial derivative of f with respect to x i at a is defined as lim

h i™0

f x iŽ a1 , a2 , . . . , a i q h i , . . . , a n . y f x iŽ a1 , a2 , . . . , a i , . . . , a n . hi

and is denoted by ⭸ 2 f Ža.r⭸ x i2 , or f x i x iŽa.. Also, the second-order partial derivative of f with respect to x i and x j , i / j, at a is given by lim

h j™0

f x iŽ a1 , a2 , . . . , a j q h j , . . . , a n . y f x iŽ a1 , a2 , . . . , a j , . . . , a n . hj

and is denoted by ⭸ 2 f Ža.r⭸ x j ⭸ x i , or f x j x iŽa., i / j.

269

DERIVATIVES OF A MULTIVARIABLE FUNCTION

Under certain conditions, the order in which differentiation with respect to x i and x j takes place is irrelevant, that is, f x i x jŽa. is identical to f x j x iŽa., i / j. This property is known as the commutative property of partial differentiation and is proved in the next theorem. Theorem 7.4.1. Let f: D ™ R m , where D ; R n, and let a be an interior point of D. Suppose that in a neighborhood of a the following conditions are satisfied: 1. ⭸ f Žx.r⭸ x i and ⭸ f Žx.r⭸ x j exist and are finite Ž i, j s 1, 2, . . . , n, i / j .. 2. Of the derivatives ⭸ 2 f Žx.r⭸ x i ⭸ x j , ⭸ 2 f Žx.r⭸ x j ⭸ x i one exists and is continuous. Then

⭸ 2 f Ž a. ⭸ xi ⭸ x j

s

⭸ 2 f Ž a. ⭸ x j ⭸ xi

.

Proof. Let us suppose that ⭸ 2 f Žx.r⭸ x j ⭸ x i exists and is continuous in a neighborhood of a. Without loss of generality we assume that i - j. If ⭸ 2 f Ža.r⭸ x i ⭸ x j exists, then it must be equal to the limit lim

f x jŽ a1 , a2 , . . . , a i q h i , . . . , a n . y f x jŽ a1 , a2 , . . . , a i , . . . , a n . hi

h i™0

,

that is, lim

h i™0

1

lim

hi

h j™0

1 hj


f Ž a1 , a2 , . . . , ai q h i , . . . , a j q h j , . . . , an . yf Ž a1 , a2 , . . . , a i q h i , . . . , a j , . . . , a n . 4

y lim

h j™0

1 hj


f Ž a1 , a2 , . . . , ai , . . . , a j q h j , . . . , an . yf Ž a1 , a2 , . . . , a i , . . . , a j , . . . , a n . 4 .

Ž 7.9 .

Let us denote f Ž x 1 , x 2 , . . . , x j q h j , . . . , x n . y f Ž x 1 , x 2 , . . . , x j , . . . , x n . by ␺ Ž x 1 , x 2 , . . . , x n .. Then the double limit in Ž7.9. can be written as lim lim

h i™0 h j™0

1 hi h j

␺ Ž a1 , a2 , . . . , a i q h i , . . . , a j , . . . , a n . y␺ Ž a1 , a2 , . . . , a i , . . . , a j , . . . , a n .

s lim lim

h i™0 h j™0

1

⭸ ␺ Ž a1 , a2 , . . . , a i q ␪ i h i , . . . , a j , . . . , a n .

hj

⭸ xi

, Ž 7.10 .

270

MULTIDIMENSIONAL CALCULUS

where 0 - ␪ i - 1. In formula Ž7.10. we have applied the mean value theorem ŽTheorem 4.2.2. to ␺ as if it were a function of the single variable x i Žsince ⭸ fr⭸ x i , and hence ⭸ ␺r⭸ x i exists in a neighborhood of a.. The right-hand side of Ž7.10. can then be written as lim lim

h i™0 h j™0

1

⭸ f Ž a1 , a2 , . . . , a i q ␪ i h i , . . . , a j q h j , . . . , a n .

hj

⭸ xi y

s lim lim

h i™0 h j™0

⭸ f Ž a1 , a2 , . . . , a i q ␪ i h i , . . . , a j , . . . , a n . ⭸ xi ⭸

⭸ f Ž a1 , a2 , . . . , a i q ␪ i h i , . . . , a j q ␪ j h j , . . . , a n .

⭸ xj

⭸ xi

,

Ž 7.11 . where 0 - ␪ j - 1. In formula Ž7.11. we have again made use of the mean value theorem, since ⭸ 2 f Žx.r⭸ x j ⭸ x i exists in the given neighborhood around a. Furthermore, since ⭸ 2 f Žx.r⭸ x j ⭸ x i is continuous in this neighborhood, the double limit in Ž7.11. is equal to ⭸ 2 f Ža.r⭸ x j ⭸ x i . This establishes the assertion that the two second-order partial derivatives of f are equal. I EXAMPLE 7.4.1. Consider the function f : R 3 ™ R, where f Ž x 1 , x 2 , x 3 . s x 1 e x 2 q x 2 cos x 1. Then

⭸ f Ž x1 , x 2 , x 3 . ⭸ x1 ⭸ f Ž x1 , x 2 , x 3 . ⭸ x2 ⭸ 2 f Ž x1 , x 2 , x 3 . ⭸ x 2 ⭸ x1 ⭸ 2 f Ž x1 , x 2 , x 3 . ⭸ x1 ⭸ x 2

s e x 2 y x 2 sin x 1 , s x 1 e x 2 q cos x 1 , s e x 2 y sin x 1 , s e x 2 y sin x 1 .

7.4.1. The Total Derivative Let f Žx. be a real-valued function defined on a set D ; R n, where x s Ž x 1 , x 2 , . . . , x n .⬘. Suppose that x 1 , x 2 , . . . , x n are functions of a single variable t. Then f is a function of t. The ordinary derivative of f with respect to t, namely dfrdt, is called the total derivative of f.

271

DERIVATIVES OF A MULTIVARIABLE FUNCTION

Let us now assume that for the values of t under consideration dx irdt exists for i s 1, 2, . . . , n and that ⭸ f Žx.r⭸ x i exists and is continuous in the interior of D for i s 1, 2, . . . , n. Under these considerations, the total derivative of f is given by df dt

n

s

Ý

is1

⭸ f Ž x . dx i ⭸ xi

dt

Ž 7.12 .

.

To show this we proceed as follows: Let ⌬ x 1 , ⌬ x 2 , . . . , ⌬ x n be increments of x 1 , x 2 , . . . , x n that correspond to an increment ⌬ t of t. In turn, f will have the increment ⌬ f. We then have ⌬ f s f Ž x1 q ⌬ x1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n . y f Ž x1 , x 2 , . . . , x n . . This can be written as ⌬ f s f Ž x1 q ⌬ x1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n . yf Ž x 1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n . q f Ž x1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n . yf Ž x 1 , x 2 , x 3 q ⌬ x 3 , . . . , x n q ⌬ x n . q f Ž x1 , x 2 , x 3 q ⌬ x 3 , . . . , x n q ⌬ x n . yf Ž x 1 , x 2 , x 3 , x 4 q ⌬ x 4 , . . . , x n q ⌬ x n . q ⭈⭈⭈ q f Ž x 1 , x 2 , . . . , x ny1 , x n q ⌬ x n . y f Ž x 1 , x 2 , . . . , x n . . By applying the mean value theorem to the difference in each bracket we obtain ⌬ f s ⌬ x1

⭸ f Ž x1 q ␪ 1 ⌬ x1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n .

q ⌬ x2 q ⌬ x3

⭸ x1 ⭸ f Ž x1 , x 2 q ␪ 2 ⌬ x 2 , x 3 q ⌬ x 3 , . . . , x n q ⌬ x n . ⭸ x2 ⭸ f Ž x1 , x 2 , x 3 q ␪ 3 ⌬ x 3 , x 4 q ⌬ x 4 , . . . , x n q ⌬ x n .

q ⭈⭈⭈ q⌬ x n

⭸ x3 ⭸ f Ž x 1 , x 2 , . . . , x ny1 , x n q ␪n ⌬ x n . ⭸ xn

,

272

MULTIDIMENSIONAL CALCULUS

where 0 - ␪ i - 1 for i s 1, 2, . . . , n. Hence, ⌬f ⌬t

s

⌬ x1 ⭸ f Ž x1 q ␪ 1 ⌬ x1 , x 2 q ⌬ x 2 , . . . , x n q ⌬ x n .

⭸ x1

⌬t q

⌬ x 2 ⭸ f Ž x1 , x 2 q ␪ 2 ⌬ x 2 , x 3 q ⌬ x 3 , . . . , x n q ⌬ x n .

q

⌬ x 3 ⭸ f Ž x1 , x 2 , x 3 q ␪ 3 ⌬ x 3 , x 4 q ⌬ x 4 , . . . , x n q ⌬ x n .

⭸ x2

⌬t

⭸ x3

⌬t

q ⭈⭈⭈ q

⌬ x n ⭸ f Ž x 1 , x 2 , . . . , x ny1 , x n q ␪n ⌬ x n .

⭸ xn

⌬t

.

Ž 7.13 .

As ⌬ t ™ 0, ⌬ x ir⌬t ™ dx irdt, and the partial derivatives in Ž7.13., being continuous, tend to ⭸ f Žx.r⭸ x i for i s 1, 2, . . . , n. Thus ⌬ fr⌬t tends to the right-hand side of formula Ž7.12.. For example, consider the function f Ž x 1 , x 2 . s x 12 y x 23 , where x 1 s t e cos t, x 2 s cos t q sin t. Then, df dt

s 2 x 1 Ž e t cos t y e t sin t . y 3 x 22 Ž ysin t q cos t . s 2 e t cos t Ž e t cos t y e t sin t . y 3 Ž cos t q sin t . Ž ysin t q cos t . 2

s Ž cos t y sin t . Ž 2 e 2 t cos t y 6 sin t cos t y 3 . . Of course, the same result could have been obtained by expressing f directly as a function of t via x 1 and x 2 and then differentiating it with respect to t. We can generalize formula Ž7.12. by assuming that each of x 1 , x 2 , . . . , x n is a function of several variables including the variable t. In this case, we need to consider the partial derivative ⭸ fr⭸ t, which can be similarly shown to have the value

⭸f ⭸t

n

s

Ý

is1

⭸ f Ž x. ⭸ x i ⭸ xi

⭸t

.

Ž 7.14 .

In general, the expression n

dfs

Ý

⭸ f Ž x.

is1

is called the total differential of f at x.

⭸ xi

dx i

Ž 7.15 .

273

DERIVATIVES OF A MULTIVARIABLE FUNCTION

EXAMPLE 7.4.2. Consider the equation f Ž x 1 , x 2 . s 0, which in general represents a relation between x 1 and x 2 . It may or may not define x 2 as a function of x 1. In this case, x 2 is said to be an implicit function of x 1. If x 2 can be obtained as a function of x 1 , then we write x 2 s g Ž x 1 .. Consequently, f w x 1 , g Ž x 1 .x will be identically equal to zero. Hence, f w x 1 , g Ž x 1 .x, being a function of one variable x 1 , will have a total derivative identically equal to zero. By applying formula Ž7.12. with t s x 1 we obtain df dx 1

s

⭸f

q

⭸ x1

⭸ f dx 2

' 0.

⭸ x 2 dx 1

If ⭸ fr⭸ x 2 / 0, then the derivative of x 2 is given by dx 2 dx 1

s

y⭸ fr⭸ x 1

⭸ fr⭸ x 2

Ž 7.16 .

.

In particular, if f Ž x 1 , x 2 . s 0 is of the form x 1 y hŽ x 2 . s 0, and if this equation can be solved uniquely for x 2 in terms of x 1 , then x 2 represents the inverse function of h, that is, x 2 s hy1 Ž x 1 .. Thus according to formula Ž7.16., dhy1 dx 1

s

1 dhrdx 2

.

This agrees with the formula for the derivative of the inverse function given in Theorem 4.2.4. 7.4.2. Directional Derivatives Let f: D ™ R m , where D ; R n, and let v be a unit vector in R n Žthat is, a vector whose length is equal to one., which represents a certain direction in the n-dimensional Euclidean space. By definition, the directional derivative of f at a point x is the interior of D in the direction of v is given by the limit lim h™0

f Ž x q hv . y f Ž x . h

,

if it exists. In particular, if v s e i , the unit vector in the direction of the ith coordinate axis, then the directional derivative of f in the direction of v is just the partial derivative of f with respect to x i Ž i s 1, 2, . . . , n.. Lemma 7.4.1. Let f: D ™ R m , where D ; R n. If the partial derivatives ⭸ f jr⭸ x i exist at a point x s Ž x 1 , x 2 , . . . , x n .⬘ in the interior of D for i s 1, 2, . . . , n; j s 1, 2, . . . , m, where f j is the jth element of f, then the directional derivative of f at x in the direction of a unit vector v exists and is equal to Jf Žx.v, where Jf Žx. is the m = n Jacobian of f at x.

274

MULTIDIMENSIONAL CALCULUS

Proof. Let us first consider the directional derivative of f j in the direction of v. To do so, we rotate the coordinate axes so that v coincides with the direction of the ␰ 1-axis, where ␰ 1 , ␰ 2 , . . . , ␰ n are the resulting new coordinates. By the well-known relations for rotation of axes in analytic geometry of n dimensions we have n

x i s ␰ 1®i q

Ý ␰ l ␭l i ,

Ž 7.17 .

i s 1, 2, . . . , n,

ls2

where ®i is the ith element of v Ž i s 1, 2, . . . , n. and ␭ l i is the ith element of ␭ l , the unit vector in the direction of the ␰ l-axis Ž l s 2, 3, . . . , n.. Now, the directional derivative of f j in the direction of v can be obtained by first expressing f j as a function of ␰ 1 , ␰ 2 , . . . , ␰ n using the relations Ž7.17. and then differentiating it with respect to ␰ 1. By formula Ž7.14., this is equal to

⭸ fj ⭸␰ 1

n

s

Ý

is1 n

s

Ý

is1

⭸ fj ⭸ xi ⭸ x i ⭸␰ 1 ⭸ fj ⭸ xi

®i ,

Ž 7.18 .

j s 1, 2, . . . , m.

From formula Ž7.18. we conclude that the directional derivative of f s Ž f 1 , f 2 , . . . , f m .⬘ in the direction of v is equal to Jf Žx.v. I EXAMPLE 7.4.3.

Let f: R 3 ™ R 2 be defined as f Ž x1 , x 2 , x 3 . s

x 12 q x 22 q x 32

.

x 12 y x 1 x 2 q x 32

The directional derivative of f at x s Ž1, 2, 1.⬘ in the direction of v s Ž1r '2 , y1r '2 , 0.⬘ is 1

2 x1 Jf Ž x . v s 2 x1 y x 2

2 x2 yx 1

2 x3 2 x3

'2 Ž1, 2, 1 .

y

1

'2 0

1

s

2 0

4 y1

2 2

y2

'2 y

1

'2 0

s

'2 1

'2

.

275

DERIVATIVES OF A MULTIVARIABLE FUNCTION

Definition 7.4.1. Let f : D ™ R, where D ; R n. If the partial derivatives ⭸ fr⭸ x i Ž i s 1, 2, . . . , n. exist at a point x s Ž x 1 , x 2 , . . . , x n .⬘ in the interior of D, then the vector Ž ⭸ fr⭸ x 1 , ⭸ fr⭸ x 2 , . . . , ⭸ fr⭸ x n .⬘ is called the gradient of f at x and is denoted by ⵱f Žx.. I Using Definition 7.4.1, the directional derivative of f at x in the direction of a unit vector v can be expressed as ⵱f Žx.⬘v, where ⵱f Žx.⬘ denotes the transpose of ⵱f Žx.. The Geometric Meaning of the Gradient Let f : D ™ R, where D ; R n. Suppose that the partial derivatives of f exist at a point x s Ž x 1 , x 2 , . . . , x n .⬘ in the interior of D. Let C denote a smooth curve that lies on the surface of f Žx. s c 0 , where c 0 is a constant, and passes through the point x. This curve can be represented by the equations x 1 s g 1Ž t ., x 2 s g 2 Ž t ., . . . , x n s g nŽ t ., where aF t F b. By formula Ž7.12., the total derivative of f with respect to t at x is df dt

n

s

Ý

is1

⭸ f Ž x . dg i ⭸ xi

dt

Ž 7.19 .

.

The vector ␭ s Ž dg 1rdt, dg 2rdt, . . . , dg nrdt .⬘ is tangent to C at x. Thus from Ž7.19. we obtain df dt

s ⵱f Ž x . ⬘␭ .

Ž 7.20 .

Now, since f w g 1Ž t ., g 2 Ž t ., . . . , g nŽ t .x s c 0 along C, then dfrdts 0 and hence ⵱f Žx.⬘␭ s 0. This indicates that the gradient vector is orthogonal to ␭ , and hence to C, at x g D. Since this result is true for any smooth curve through x, we conclude that the gradient vector ⵱f Žx. is orthogonal to the surface of f Žx. s c 0 at x. Definition 7.4.2. Let f : D ™ R, where D ; R n. Then ⵱f : D ™ R n. The Jacobian matrix of ⵱f Žx. is called the Hessian matrix of f and is denoted by H f Žx.. Thus H f Žx. s J ⵱ f Žx., that is,

H f Ž x. s

⭸ 2 f Ž x.

⭸ 2 f Ž x.

⭸ x 12 . . . 2 ⭸ f Ž x.

⭸ x 2 ⭸ x1 . . . 2 ⭸ f Ž x.

⭸ x1 ⭸ x n

⭸ x2 ⭸ xn

⭈⭈⭈

⭈⭈⭈

⭸ 2 f Ž x. ⭸ x n ⭸ x1 . . . 2 ⭸ f Ž x. ⭸ x n2

.

Ž 7.21 .

276

MULTIDIMENSIONAL CALCULUS

The determinant of H f Žx. is called the Hessian determinant. If the conditions of Theorem 7.4.1 regarding the commutative property of partial differentiation are valid, then H f Žx. is a symmetric matrix. As we shall see in Section 7.7, the Hessian matrix plays an important role in the identification of maxima and minima of a multivariable function. I 7.4.3. Differentiation of Composite Functions Let f: D 1 ™ R m , where D 1 ; R n, and let g: D 2 ™ R p , where D 2 ; R m. Let x 0 be an interior point of D 1 and f Žx 0 . be an interior point of D 2 . If the m = n Jacobian matrix Jf Žx 0 . and the p= m Jacobian matrix J g wf Žx 0 .x both exist, then the p= n Jacobian matrix J hŽx 0 . for the composite function h s g(f exists and is given by J h Ž x 0 . s J g f Ž x 0 . Jf Ž x 0 . .

Ž 7.22 .

To prove formula Ž7.22., let us consider the Ž k, i .th element of J hŽx 0 ., namely ⭸ h k Žx 0 .r⭸ x i , where h k Žx 0 . s g k wf Žx 0 .x is the kth element of hŽx 0 . s gwf Žx 0 .x, i s 1, 2, . . . , n; k s 1, 2, . . . , p, By applying formula Ž7.14. we obtain

⭸ hk Ž x 0 . ⭸ xi

m

s

Ý

js1

⭸ gk f Ž x 0 . ⭸ fj Ž x 0 . ⭸ fj

⭸ xi

,

i s 1, 2, . . . , n; k s 1, 2, . . . , p,

Ž 7.23 . where f j Žx 0 . is the jth element of f Žx 0 ., j s 1, 2, . . . , m. But ⭸ g k wf Žx 0 .xr⭸ f j is the Ž k, j .th element of J g wf Žx 0 .x, and ⭸ f j Žx 0 .r⭸ x i is the Ž j, i .th element of Jf Žx 0 ., i s 1, 2, . . . , n; j s 1, 2, . . . , m; k s 1, 2, . . . , p. Hence, formula Ž7.22. follows from formula Ž7.23. and the rule of matrix multiplication. In particular, if m s n s p, then from formula Ž7.22., the determinant of Ž J h x 0 . is given by det J h Ž x 0 . s det J g Ž f Ž x 0 . . det Jf Ž x 0 . .

Ž 7.24 .

Using the notation in formula Ž7.8., formula Ž7.24. can be expressed as

⭸ Ž h1 , h 2 , . . . , h n . ⭸ Ž x1 , x 2 , . . . , x n . EXAMPLE 7.4.4.

s

⭸ Ž g1 , g 2 , . . . , g n . ⭸ Ž f1 , f 2 , . . . , fn . ⭸ Ž f1 , f 2 , . . . , fn . ⭸ Ž x1 , x 2 , . . . , x n .

Let f: R 2 ™ R 3 be given by x 12 y x 2 cos x 1 x1 x 2 f Ž x1 , x 2 . s . x 13 q x 23

.

Ž 7.25 .

TAYLOR’S THEOREM FOR A MULTIVARIABLE FUNCTION

277

Let g: R 3 ™ R be defined as g Ž ␰ 1 , ␰ 2 , ␰ 3 . s ␰ 1 y ␰ 22 q ␰ 3 , where

␰ 1 s x 12 y x 2 cos x 1 , ␰ 2 s x1 x 2 , ␰ 3 s x 13 q x 23 . In this case, 2 x 1 q x 2 sin x 1 x2 Jf Ž x . s 3 x 12

ycos x 1 x1 , 2 3 x2

J g f Ž x . s Ž 1,y 2 ␰ 2 , 1 . . Hence, by formula Ž7.22., 2 x 1 q x 2 sin x 1 x2 J h Ž x . s Ž 1,y 2 ␰ 2 , 1 . 3 x 12

ycos x 1 x1 3 x 22

s Ž 2 x 1 q x 2 sin x 1 y 2 x 1 x 22 q 3 x 12 ,y cos x 1 y 2 x 12 x 2 q 3 x 22 . . 7.5. TAYLOR’S THEOREM FOR A MULTIVARIABLE FUNCTION We shall now consider a multidimensional analogue of Taylor’s theorem, which was discussed in Section 4.3 for a single-variable function. Let us first introduce the following notation: Let x s Ž x 1 , x 2 , . . . , x n .⬘. Then x⬘⵱ denotes a first-order differential operator of the form

⭸

n

x⬘⵱ s

Ý xi ⭸ x

is1

. i

The symbol ⵱, called the del operator, was used earlier to define the gradient vector. If m is a positive integer, then Žx⬘⵱ . m denotes an mth-order differential operator. For example, for m s n s 2,

ž

Ž x⬘⵱ . s x 1 2

s x 12

⭸ ⭸ x1

⭸2 ⭸ x 12

q x2

⭸ ⭸ x2

q 2 x1 x 2

/

2

⭸2 ⭸ x1 ⭸ x 2

q x 22

⭸2 ⭸ x 22

.

278

MULTIDIMENSIONAL CALCULUS

Thus Žx⬘⵱ . 2 is obtained by squaring x 1 ⭸r⭸ x 1 q x 2 ⭸r⭸ x 2 in the usual fashion, except that the squares of ⭸r⭸ x 1 and ⭸r⭸ x 2 are replaced by ⭸ 2r⭸ x 12 and ⭸ 2r⭸ x 22 , respectively, and the product of ⭸r⭸ x 1 and ⭸r⭸ x 2 is replaced by ⭸ 2r⭸ x 1 ⭸ x 2 Žhere we are assuming that the commutative property of partial differentiation holds once these differential operators are applied to a real-valued function.. In general, Žx⬘⵱ . m is obtained by a multinomial expansion of degree m of the form m

Ž x⬘⵱ . s

Ý

k1 , k 2 , . . . , k n

ž

/

⭸m m k1 k 2 kn x x ⭈⭈⭈ x , n k1 , k 2 , . . . , k n 1 2 ⭸ x 1k 1 ⭸ x 2k 2 ⭈⭈⭈ ⭸ x nk n

where the sum is taken over all n-tuples Ž k 1 , k 2 , . . . , k n . for which Ý nis1 k i s m, and

ž

/

m! m k 1 , k 2 , . . . , k n s k 1 ! k 2 ! ⭈⭈⭈ k n ! .

If a real-valued function f Žx. has partial derivatives through order m, then an application of the differential operator Žx⬘⵱ . m to f Žx. results in m

Ž x⬘⵱ . f Ž x . s

ž

Ý

k1 , k 2 , . . . , k n

/

m k1 k 2 k 1 , k 2 , . . . , k n x 1 x 2 ⭈⭈⭈

= x nk n

⭸ m f Ž x. ⭸ x 1k 1 ⭸ x 2k 2 ⭈⭈⭈ ⭸ x nk n

Ž 7.26 .

.

The notation Žx⬘⵱ . m f Žx 0 . indicates that Žx⬘⵱ . m f Žx. is evaluated at x 0 . Theorem 7.5.1. Let f : D ™ R, where D ; R n, and let N␦ Žx 0 . be a neighborhood of x 0 g D such that N␦ Žx 0 . ; D. If f and all its partial derivatives of order F r exist and are continuous in N␦ Žx 0 ., then for any x g N␦ Žx 0 ., f Ž x. s f Ž x 0 . q

ry1

Ý

is1

Ž x y x 0 . ⬘⵱ f Ž x 0 .

r

i

i!

q

Ž x y x 0 . ⬘⵱ f Ž z 0 . r!

, Ž 7.27 .

where z 0 is a point on the line segment from x 0 to x. Proof. Let h s x y x 0 . Let the function ␾ Ž t . be defined as ␾ Ž t . s f Žx 0 q t h., where 0 F t F 1. If t s 0, then ␾ Ž0. s f Žx 0 . and ␾ Ž1. s f Žx 0 q h. s f Žx., if

TAYLOR’S THEOREM FOR A MULTIVARIABLE FUNCTION

279

t s 1. Now, by formula Ž7.12., d␾ Ž t . dt

⭸ f Ž x.

n

Ý hi

s

⭸ xi

is1

xsx 0 qt h

s Ž h⬘⵱ . f Ž x 0 q t h . , where h i is the ith element of h Ž i s 1, 2, . . . , n.. Furthermore, the derivative of order m of ␾ Ž t . is d m␾ Ž t . dt

m

m

s Ž h⬘⵱ . f Ž x 0 q t h . ,

1FmFr.

Since the partial derivatives of f through order r are continuous, then the same order derivatives of ␾ Ž t . are also continuous on w0, 1x and d m␾ Ž t . dt m

m

ts0

s Ž h⬘⵱ . f Ž x 0 . ,

1FmFr.

If we now apply Taylor’s theorem in Section 4.3 to the single-variable function ␾ Ž t ., we obtain

␾ Ž t . s ␾ Ž 0. q

ry1

Ý

is1

i t i d␾ Ž t.

dt i

i!

q ts0

t r d r␾ Ž t . dt r

r!

ts␰

Ž 7.28 .

,

where 0 - ␰ - t. By setting t s 1 in formula Ž7.28., we obtain f Ž x. s f Ž x 0 . q

ry1

Ý

Ž x y x 0 . ⬘⵱ f Ž x 0 .

is1

r

i

i!

q

Ž x y x 0 . ⬘⵱ f Ž z 0 . r!

,

where z 0 s x 0 q ␰ h. Since 0 - ␰ - 1, the point z 0 lies on the line segment between x 0 and x. I In particular, if f Žx. has partial derivatives of all orders in N␦ Žx 0 ., then we have the series expansion f Ž x. s f Ž x 0 . q

⬁

Ý

is1

Ž x y x 0 . ⬘⵱ f Ž x 0 . i

i!

.

Ž 7.29 .

In this case, the last term in formula Ž7.27. serves as a remainder of Taylor’s series.

280

MULTIDIMENSIONAL CALCULUS

EXAMPLE 7.5.1. Consider the function f : R 2 ™ R defined as f Ž x 1 , x 2 . s x 1 x 2 q x 12 q e x 1 cos x 2 . This function has partial derivatives of all orders. Thus in a neighborhood of x 0 s Ž0, 0.⬘ we can write f Ž x 1 , x 2 . s 1 q Ž x⬘⵱ . f Ž 0, 0 . q

1 2!

2 Ž x⬘⵱ . f Ž 0, 0 . q

1 3!

3 Ž x⬘⵱ . f Ž ␰ x 1 , ␰ x 2 . ,

0 - ␰ - 1. It can be verified that

Ž x⬘⵱ . f Ž 0, 0 . s x 1 , 2 Ž x⬘⵱ . f Ž 0, 0 . s 3 x 12 q 2 x 1 x 2 y x 22 , 3 Ž x⬘⵱ . f Ž ␰ x 1 , ␰ x 2 . s x 13 e ␰ x 1 cos Ž ␰ x 2 . y 3 x 12 x 2 e ␰ x 1 sin Ž ␰ x 2 .

y 3 x 1 x 22 e ␰ x 1 cos Ž ␰ x 2 . q x 23 e ␰ x 1 sin Ž ␰ x 2 . . Hence, f Ž x1 , x 2 . s 1 q x1 q q

1 3!




1 2!

Ž 3 x 12 q 2 x 1 x 2 y x 22 .

x 13 y 3 x 1 x 22 e ␰ x 1 cos Ž ␰ x 2 . q x 23 y 3 x 12 x 2 e ␰ x 1 sin Ž ␰ x 2 . 4 .

The first three terms serve as a second-order approximation of f Ž x 1 , x 2 ., while the last term serves as a remainder.

7.6. INVERSE AND IMPLICIT FUNCTION THEOREMS Consider the function f: D ™ R n, where D ; R n. Let y s f Žx.. The purpose of this section is to present conditions for the existence of an inverse function fy1 which expresses x as a function of y. These conditions are given in the next theorem, whose proof can be found in Sagan Ž1974, page 371.. See also Fulks Ž1978, page 346.. Theorem 7.6.1 ŽInverse Function Theorem.. Let f: D ™ R n, where D is an open subset of R n and f has continuous first-order partial derivatives in D. If for some x 0 g D, the n = n Jacobian matrix Jf Žx 0 . is nonsingular,

281

INVERSE AND IMPLICIT FUNCTION THEOREMS

that is,

⭸ Ž f1 , f 2 , . . . , fn .

det Jf Ž x 0 . s

⭸ Ž x1 , x 2 , . . . , x n .

/ 0, xsx 0

where f i is the ith element of f Ž i s 1, 2, . . . , n., then there exist an ⑀ ) 0 and a ␦ ) 0 such that an inverse function fy1 exists in the neighborhood N␦ wf Žx 0 .x and takes values in the neighborhood N⑀ Žx 0 .. Moreover, fy1 has continuous first-order partial derivatives in N␦ wf Žx 0 .x, and its Jacobian matrix at f Žx 0 . is the inverse of Jf Žx 0 .; hence, det
Jfy1 f Ž x 0 . EXAMPLE 7.6.1.

4s

1 det Jf Ž x 0 .

Ž 7.30 .

.

Let f: R 3 ™ R 3 be given by 2 x1 x 2 y x 2 f Ž x1 , x 2 , x 3 . s

x 12 q x 2 q 2

x 32 .

x1 x 2 q x 2 Here, 2 x2 Jf Ž x . s 2 x 1 x2

2 x1 y 1 1 x1 q 1

0 4 x3 , 0

and detwJf Žx.x s y12 x 2 x 3 . Hence, all x g R 3 at which x 2 x 3 / 0, f has an inverse function fy1 . For example, if D s
Ž x 1 , x 2 , x 3 .< x 2 ) 0, x 3 ) 04 , then f is invertible in D. From the equations y1 s 2 x1 x 2 y x 2 , y 2 s x 12 q x 2 q 2 x 32 , y3 s x1 x 2 q x 2 , we obtain the inverse function x s fy1 Žy., where x1 s x2 s x3 s

y1 q y 3 2 y 3 y y1

,

yy 1 q 2 y 3 3 1

'2

y2 y

,

Ž y1 q y 3 .

2

Ž 2 y 3 y y1 .

2

y

2 y 3 y y1 3

1r2

.

282

MULTIDIMENSIONAL CALCULUS

If, for example, we consider x 0 s Ž1, 1, 1.⬘, then y0 s f Žx 0 . s Ž1, 4, 2.⬘, and detwJf Žx 0 .x s y12. The Jacobian matrix of fy1 at y0 is 2 3

0

y 13

1 Jfy1 Ž y0 . s y 3

0

2 3

1 4

0

y

1 4

.

Its determinant is equal to det Jfy1 Ž y0 . s y 121 . We note that this is the reciprocal of detwJf Žx 0 .x, as it should be according to formula Ž7.30.. The inverse function theorem can be viewed as providing a unique solution to a system of n equations given by y s f Žx.. There are, however, situations in which y is not explicitly expressed as a function of x. In general, we may have two vectors, x and y, of orders n = 1 and m = 1, respectively, that satisfy the relation g Ž x, y . s 0,

Ž 7.31 .

where g: R mqn ™ R n. In this more general case, we have n equations involving m q n variables, namely, the elements of x and those of y. The question now is what conditions will allow us to solve equations Ž7.31. uniquely for x in terms of y. The answer to this question is given in the next theorem, whose proof can be found in Fulks Ž1978, page 352.. Theorem 7.6.2 ŽImplicit Function Theorem.. Let g: D ™ R n, where D is an open subset of R mqn , and g has continuous first-order partial derivatives in D. If there is a point z 0 g D, where z 0 s Žx X0 , y0X .⬘ with x 0 g R n, y0 g R m such that gŽz 0 . s 0, and if at z 0 ,

⭸ Ž g1 , g2 , . . . , gn . ⭸ Ž x1 , x 2 , . . . , x n .

/ 0,

where g i is the ith element of g Ž i s 1, 2, . . . , n., then there is a neighborhood N␦ Žy0 . of y0 in which the equation gŽx, y. s 0 can be solved uniquely for x as a continuously differentiable function of y.

283

OPTIMA OF A MULTIVARIABLE FUNCTION

Let g: R 3 ™ R 2 be given by

EXAMPLE 7.6.2.

gŽ x1 , x 2 , y . s

x 1 q x 2 q y 2 y 18 . x1 y x1 x 2 q y y 4

We have

⭸ Ž g1 , g2 . ⭸ Ž x1 , x 2 .

s det

ž

1 1 y x2

/

1 yx 1

s x 2 y x 1 y 1.

Let z s Ž x 1 , x 2 , y .⬘. At the point z 0 s Ž1, 1, 4.⬘, for example, gŽz 0 . s 0 and ⭸ Ž g 1 , g 2 .r⭸ Ž x 1 , x 2 . s y1 / 0. Hence, by Theorem 7.6.2, we can solve the equations x 1 q x 2 q y 2 y 18 s 0,

Ž 7.32 .

x1 y x1 x 2 q y y 4 s 0

Ž 7.33 .

uniquely in terms of y in some neighborhood of y 0 s 4. For example, if D in Theorem 7.6.2 is of the form D s
Ž x 1 , x 2 , y . < x 1 ) 0, x 2 ) 0, y - 4.06 4 , then from equations Ž7.32. and Ž7.33. we obtain the solution

Ž y y 17 . y 4 y q 16 5 , ½ ½ 19 y y y Ž y y 17. y 4 y q 16 5 .

x 1 s 12 y Ž y 2 y 17 . q x 2 s 12

2

2

1r2

2

2

2

1r2

We note that the sign preceding the square root in the formula for x 1 was chosen as q, so that x 1 s x 2 s 1 when y s 4. It can be verified that Ž y 2 y 17. 2 y 4 y q 16 is positive for y - 4.06.

7.7. OPTIMA OF A MULTIVARIABLE FUNCTION Let f Žx. be a real-valued function defined on a set D ; R n. A point x 0 g D is said to be a point of local maximum of f if there exists a neighborhood N␦ Žx 0 . ; D such that f Žx. F f Žx 0 . for all x g N␦ Žx 0 .. If f Žx. G f Žx 0 . for all x g N␦ Žx 0 ., then x 0 is a point of local minimum. If one of these inequalities holds for all x in D, then x 0 is called a point of absolute maximum, or a point of absolute minimum, respectively, of f in D. In either case, x 0 is referred to as a point of optimum Žor extremum., and the value of f Žx. at x s x 0 is called an optimum value of f Žx..

284

MULTIDIMENSIONAL CALCULUS

In this section we shall discuss conditions under which f Žx. attains local optima in D. Then, we shall investigate the determination of the optima of f Žx. over a constrained region of D. As in the case of a single-variable function, if f Žx. has first-order partial derivatives at a point x 0 in the interior of D, and if x 0 is a point of local optimum, then ⭸ fr⭸ x i s 0 for i s 1, 2, . . . , n at x 0 . The proof of this fact is similar to that of Theorem 4.4.1. Thus the vanishing of the first-order partial derivatives of f Žx. at x 0 is a necessary condition for a local optimum at x 0 , but is obviously not sufficient. The first-order partial derivatives can be zero without necessarily having a local optimum at x 0 . In general, any point at which ⭸ fr⭸ x i s 0 for i s 1, 2, . . . , n is called a stationary point. It follows that any point of local optimum at which f has first-order partial derivatives is a stationary point, but not every stationary point is a point of local optimum. If no local optimum is attained at a stationary point x 0 , then x 0 is called a saddle point. The following theorem gives the conditions needed to have a local optimum at a stationary point. Theorem 7.7.1. Let f : D ™ R, where D ; R n. Suppose that f has continuous second-order partial derivatives in D. If x 0 is a stationary point of f, then at x 0 f has the following: i. A local minimum if Žh⬘⵱ . 2 f Žx 0 . ) 0 for all h s Ž h1 , h 2 , . . . , h n .⬘ in a neighborhood of 0, where the elements of h are not all equal to zero. ii. A local maximum if Žh⬘⵱ . 2 f Žx 0 . - 0, where h is the same as in Ži.. iii. A saddle point if Žh⬘⵱ . 2 f Žx 0 . changes sign for values of h in a neighborhood of 0. Proof. By applying Taylor’s theorem to f Žx. in a neighborhood of x 0 we obtain f Ž x 0 q h . s f Ž x 0 . q Ž h⬘⵱ . f Ž x 0 . q

1 2!

2 Ž h⬘⵱ . f Ž z 0 . ,

where h is a nonzero vector in a neighborhood of 0 and z 0 is a point on the line segment from x 0 to x 0 q h. Since x 0 is a stationary point, then Žh⬘⵱ . f Žx 0 . s 0. Hence, f Ž x 0 q h. y f Ž x 0 . s

1 2!

2 Ž h⬘⵱ . f Ž z 0 . .

Also, since the second-order partial derivatives of f are continuous at x 0 , then we can write f Ž x 0 q h. y f Ž x 0 . s

1 2!

2 Ž h⬘⵱ . f Ž x 0 . q o Ž 5 h 5 . ,

285

OPTIMA OF A MULTIVARIABLE FUNCTION

where 5 h 5 s Žh⬘h.1r2 and oŽ5 h 5. ™ 0 as h ™ 0. We note that for small values of 5 h 5, the sign of f Žx 0 q h. y f Žx 0 . depends on the value of Žh⬘⵱ . 2 f Žx 0 .. It follows that if i. Žh⬘⵱ . 2 f Žx 0 . ) 0, then f Žx 0 q h. ) f Žx 0 . for all nonzero values of h in some neighborhood of 0. Thus x 0 is a point of local minimum of f. ii. Žh⬘⵱ . 2 f Žx 0 . - 0, then f Žx 0 q h. - f Žx 0 . for all nonzero values of h in some neighborhood of 0. In this case, x 0 is a point of local maximum of f. iii. Žh⬘⵱ . 2 f Žx 0 . changes sign inside a neighborhood of 0, then x 0 is neither a point of local maximum nor a point of local minimum. Therefore, x 0 must be a saddle point. I We note that Žh⬘⵱ . 2 f Žx 0 . can be written as a quadratic form of the form h⬘Ah, where A s H f Žx 0 . is the n = n Hessian matrix of f evaluated at x 0 , that is, f 11 f 21 As . . . f n1

⭈⭈⭈ ⭈⭈⭈

f 12 f 22 . . . f n2

⭈⭈⭈

f1 n f2 n . , . . fn n

Ž 7.34 .

where for simplicity we have denoted ⭸ 2 f Žx 0 .r⭸ x i ⭸ x j by f i j , i, j s 1, 2, . . . , n wsee formula Ž7.21.x. Corollary 7.7.1. Let f be the same function as in Theorem 7.7.1, and let A be the matrix given by formula Ž7.34.. If x 0 is a stationary point of f, then at x 0 f has the following: i. A local minimum if A is positive definite, that is, the leading principal minors of A Žsee Definition 2.3.6. are all positive, f 11 ) 0,

det

ž

f 11 f 21

/

f 12 f 22

) 0, . . . ,

det Ž A . ) 0.

Ž 7.35 .

ii. A local maximum if A is negative definite, that is, the leading principal minors of A have alternating signs as follows: f 11 - 0,

det

ž

f 11 f 21

f 12 f 22

/

n

) 0, . . . , Ž y1 . det Ž A . ) 0. Ž 7.36 .

iii. A saddle point if A is neither positive definite nor negative definite.

286

MULTIDIMENSIONAL CALCULUS

Proof. i. By Theorem 7.7.1, f has a local minimum at x 0 if Žh⬘⵱ . 2 f Žx 0 . s h⬘Ah is positive for all h / 0, that is, if A is positive definite. By Theorem 2.3.12Ž2., A is positive definite if and only if its leading principal minors are all positive. The conditions stated in Ž7.35. are therefore sufficient for a local minimum at x 0 . ii. Žh⬘⵱ . 2 f Žx 0 . - 0 if and only if A is negative definite, or yA is positive definite. Now, a leading principal minor of order m Žs 1, 2, . . . , n. of yA is equal to Žy1. m multiplied by the corresponding leading principal minor of A. This leads to conditions Ž7.36.. iii. If A is neither positive definite nor negative definite, then Žh⬘⵱ . 2 f Žx 0 . must change sign inside a neighborhood of x 0 . This makes x 0 a saddle point. I A Special Case If f is a function of only n s 2 variables, x 1 and x 2 , then conditions Ž7.35. and Ž7.36. can be written as: i. f 11 ) 0, f 11 f 22 y f 122 ) 0 for a local minimum at x 0 . ii. f 11 - 0, f 11 f 22 y f 122 ) 0 for a local maximum at x 0 . If f 11 f 22 y f 122 - 0, then x 0 is a saddle point, since in this case h⬘Ah s h12 s

⭸ 2 f Žx0 . ⭸ x 12

⭸ 2 f Žx0 . ⭸ x 12

q 2 h1 h 2

⭸ 2 f Žx0 . ⭸ x1 ⭸ x 2

q h 22

⭸ 2 f Žx0 . ⭸ x 22

Ž h1 y ah 2 . Ž h1 y bh 2 . ,

where ah 2 and bh 2 are the real roots of the equation h⬘Ah s 0 with respect to h1. Hence, h⬘Ah changes sign in a neighborhood of 0. If f 11 f 22 y f 122 s 0, then h⬘Ah can be written as h⬘Ah s

⭸ 2 f Žx0 . ⭸ x 12

h1 q h 2

⭸ 2 f Ž x 0 . r⭸ x 1 ⭸ x 2

2

⭸ 2 f Ž x 0 . r⭸ x 12

provided that ⭸ 2 f Žx 0 .r⭸ x 12 / 0. Thus h⬘Ah has the same sign as that of ⭸ 2 f Žx 0 .r⭸ x 12 except for those values of h s Ž h1 , h 2 .⬘ for which h1 q h 2

⭸ 2 f Ž x 0 . r⭸ x 1 ⭸ x 2 ⭸ 2 f Ž x 0 . r⭸ x 12

s 0,

287

OPTIMA OF A MULTIVARIABLE FUNCTION

in which case it is zero. In the event ⭸ 2 f Žx 0 .r⭸ x 12 s 0, then ⭸ 2 f Žx 0 .r⭸ x 1 ⭸ x 2 s 0, and h⬘Ah s h 22 ⭸ 2 f Žx 0 .r⭸ x 22 , which has the same sign as that of ⭸ 2 f Žx 0 .r⭸ x 22 , if it is different from zero, except for those values of h s Ž h1 , h 2 .⬘ for which h 2 s 0, where it is zero. It follows that when f 11 f 22 y f 122 s 0, h⬘Ah has a constant sign for all h inside a neighborhood of 0. However, it can vanish for some nonzero values of h. For such values of h, the sign of f Žx 0 q h. y f Žx 0 . depends on the signs of higher-order partial derivatives Žhigher than second order. of f at x 0 . These can be obtained from Taylor’s expansion. In this case, no decision can be made regarding the nature of the stationary point until these higher-order partial derivatives Žif they exist. have been investigated. EXAMPLE 7.7.1. Let f : R 2 ™ R be the function f Ž x 1 , x 2 . s x 12 q 2 x 22 y x 1. Consider the equations

⭸f ⭸ x1 ⭸f ⭸ x2

s 2 x 1 y 1 s 0, s 4 x 2 s 0.

The only solution is x 0 s Ž0.5, 0.⬘. The Hessian matrix is As

2 0

0 , 4

which is positive definite, since 2 ) 0 and detŽA. s 8 ) 0. The point x 0 is therefore a local minimum. Since it is the only one in R 2 , it must also be the absolute minimum. EXAMPLE 7.7.2.

Consider the function f : R 3 ™ R, where

f Ž x 1 , x 2 , x 3 . s 13 x 13 q 2 x 22 q x 32 y 2 x 1 x 2 q 3 x 1 x 3 q x 2 x 3 y 10 x 1 q 4 x 2 y 6 x 3 q 1. A stationary point must satisfy the equations

⭸f ⭸ x1 ⭸f ⭸ x2 ⭸f ⭸ x3

s x 12 y 2 x 2 q 3 x 3 y 10 s 0,

Ž 7.37 .

s y2 x 1 q 4 x 2 q x 3 q 4 s 0,

Ž 7.38 .

s 3 x 1 q x 2 q 2 x 3 y 6 s 0.

Ž 7.39 .

288

MULTIDIMENSIONAL CALCULUS

From Ž7.38. and Ž7.39. we get x 2 s x 1 y 2, x 3 s 4 y 2 x1 . By substituting these expressions in equation Ž7.37. we obtain x 12 y 8 x 1 q 6 s 0. This equation has two solutions, namely, 4 y '10 and 4 q '10 . We therefore have two stationary points,

' ' ' x Ž1. 0 s Ž 4 q 10 , 2 q 10 ,y 4 y 2 10 . ⬘, ' ' ' x Ž2. 0 s Ž 4 y 10 , 2 y 10 ,y 4 q 2 10 . ⬘. Now, the Hessian matrix is 2 x1 A s y2 3

y2 4 1

3 1 . 2

Its leading principal minors are 2 x 1 , 8 x 1 y 4, and 14 x 1 y 56. The last one is Ž1. the determinant of A. At x Ž1. 0 all three are positive. Therefore, x 0 is a point Ž2. of local minimum. At x 0 the values of the leading principal minors are 1.675, 2.7018, and y44.272. In this case, A is neither positive definite over negative definite. Thus x Ž2. 0 is a saddle point. 7.8. THE METHOD OF LAGRANGE MULTIPLIERS This method, which is due to Joseph Louis de Lagrange Ž1736᎐1813., is used to optimize a real-valued function f Ž x 1 , x 2 , . . . , x n ., where x 1 , x 2 , . . . , x n are subject to m Ž- n. equality constraints of the form g 1 Ž x 1 , x 2 , . . . , x n . s 0, g 2 Ž x 1 , x 2 , . . . , x n . s 0, . . .

Ž 7.40 .

g m Ž x 1 , x 2 , . . . , x n . s 0, where g 1 , g 2 , . . . , g m are differentiable functions. The determination of the stationary points in this constrained optimization problem is done by first considering the function F Ž x. s f Ž x. q

m

Ý ␭ j g j Ž x. ,

js1

Ž 7.41 .

289

THE METHOD OF LAGRANGE MULTIPLIERS

where x s Ž x 1 , x 2 , . . . , x n .⬘ and ␭1 , ␭2 , . . . , ␭ m are scalars called Lagrange multipliers. By differentiating Ž7.41. with respect to x 1 , x 2 , . . . , x n and equating the partial derivatives to zero we obtain

⭸F ⭸ xi

s

⭸f ⭸ xi

⭸ gj

m

q

Ý ␭j ⭸ x

js1

s 0,

i s 1, 2, . . . , n.

Ž 7.42 .

i

Equations Ž7.40. and Ž7.42. consist of m q n equations in m q n unknowns, namely, x 1 , x 2 , . . . , x n ; ␭1 , ␭2 , . . . , ␭ m . The solutions for x 1 , x 2 , . . . , x n determine the locations of the stationary points. The following argument explains why this is the case: Suppose that in equation Ž7.40. we can solve for m x i ’s, for example, x 1 , x 2 , . . . , x m , in terms of the remaining n y m variables. By Theorem 7.6.2, this is possible whenever

⭸ Ž g1 , g2 , . . . , gm . ⭸ Ž x1 , x 2 , . . . , x m .

/ 0.

Ž 7.43 .

In this case, we can write x 1 s h1 Ž x mq1 , x mq2 , . . . , x n . , x 2 s h 2 Ž x mq1 , x mq2 , . . . , x n . , . . .

Ž 7.44 .

x m s h m Ž x mq1 , x mq2 , . . . , x n . . Thus f Žx. is a function of only n y m variables, namely, x mq1 , x mq2 , . . . , x n . If the partial derivatives of f with respect to these variables exist and if f has a local optimum, then these partial derivatives must necessarily vanish, that is,

⭸f ⭸ xi

m

q

Ý

js1

⭸ f ⭸ hj ⭸ hj ⭸ xi

s 0,

i s m q 1, m q 2, . . . , n.

Ž 7.45 .

Now, if equations Ž7.44. are used to substitute h1 , h 2 , . . . , h m for x 1 , x 2 , . . . , x m , respectively, in equation Ž7.40., then we obtain the identities g 1 Ž h1 , h 2 , . . . , h m , x mq1 , x mq2 , . . . , x n . ' 0, g 2 Ž h1 , h 2 , . . . , h m , x mq1 , x mq2 , . . . , x n . ' 0, . . . g m Ž h1 , h 2 , . . . , h m , x mq1 , x mq2 , . . . , x n . ' 0.

290

MULTIDIMENSIONAL CALCULUS

By differentiating these identities with respect to x mq1 , x mq2 , . . . , x n we obtain

⭸ gk ⭸ xi

m

q

Ý

js1

⭸ gk ⭸ h j ⭸ hj ⭸ xi

i s m q 1, m q 2, . . . , n; k s 1, 2, . . . , m. Ž 7.46 .

s 0,

Let us now define the vectors ␦k s

ž ž ž ž ž

⭸ gk

⭸ gk

⭸ gk

, ,..., ⭸ x mq1 ⭸ x mq2 ⭸ xn

⭸ gk ⭸ gk ⭸ gk ␥k s , ,..., ⭸ h1 ⭸ h 2 ⭸ hm ␩ js ␺s

␶s

⭸ hj

/

⭸ hj

,

⭸ x mq1 ⭸ x mq2

,...,

⭸f

⭸f

⭸f

, ,..., ⭸ h1 ⭸ h 2 ⭸ hm

⭸f ⭸ xn

/

k s 1, 2, . . . , m,

k s 1, 2, . . . , m,

,

⭸ hj

⭸f

,

X

, ,..., ⭸ x mq1 ⭸ x mq2 ⭸ xn

⭸f

/

X

/ /

X

,

j s 1, 2, . . . , m,

X

,

X

.

Equations Ž7.45. and Ž7.46. can then be written as

w ␦ 1 : ␦ 2 : ⭈⭈⭈ : ␦ m x q w ␩ 1 : ␩ 2 : ⭈⭈⭈ : ␩ m x ⌫ s 0,

Ž 7.47 .

␺ q w ␩ 1 : ␩ 2 : ⭈⭈⭈ : ␩ m x ␶ s 0,

Ž 7.48 .

where ⌫ s w ␥ 1: ␥ 2 : ⭈⭈⭈ : ␥ m x, which is a nonsingular m = m matrix if condition Ž7.43. is valid. From equation Ž7.47. we have

w ␩ 1 : ␩ 2 : ⭈⭈⭈ : ␩ m x s y w ␦ 1 : ␦ 2 : ⭈⭈⭈ : ␦ m x ⌫y1 . By making the proper substitution in equation Ž7.48. we obtain ␺ q w ␦ 1 : ␦ 2 : ⭈⭈⭈ : ␦ m x ␭ s 0,

Ž 7.49 .

␭ s y⌫y1 ␶ .

Ž 7.50 .

where

291

THE METHOD OF LAGRANGE MULTIPLIERS

Equations Ž7.49. can then be expressed as

⭸f ⭸ xi

⭸ gj

m

q

Ý ␭j ⭸ x

js1

s 0,

i s m q 1, m q 2, . . . , n.

Ž 7.51 .

i s 1, 2, . . . , m.

Ž 7.52 .

i

From equation Ž7.50. we also have

⭸f ⭸ xi

m

q

⭸ gj

Ý ␭j ⭸ x

js1

s 0,

i

Equations Ž7.51. and Ž7.52. can now be combined into a single vector equation of the form ⵱f Ž x . q

m

Ý ␭ j ⵱g j s 0,

js1

which is the same as equation Ž7.42.. We conclude that at a stationary point of f, the values of x 1 , x 2 , . . . , x n and the corresponding values of ␭1 , ␭2 , . . . , ␭ m must satisfy equations Ž7.40. and Ž7.42.. Sufficient Conditions for a Local Optimum in the Method of Lagrange Multipliers Equations Ž7.42. are only necessary for a stationary point x 0 to be a point of local optimum of f subject to the constraints given by equations Ž7.40.. Sufficient conditions for a local optimum are given in Gillespie Ž1954, pages 97᎐98.. The following is a reproduction of these conditions: Let x 0 be a stationary point of f whose coordinates satisfy equations Ž7.40. and Ž7.42., and let ␭1 , ␭2 , . . . , ␭ m be the corresponding Lagrange multipliers. Let Fi j denote the second-order partial derivative of F in formula Ž7.41. with respect to x i , and x j , i, j s 1, 2, . . . , n; i / j. Consider the Ž m q n. = Ž m q n. matrix

B1 s

F11

F12

⭈⭈⭈

F1 n

g 1Ž1.

g 2Ž1.

⭈⭈⭈

g mŽ1.

F21 . . . Fn1

F22 . . . Fn2

⭈⭈⭈

g 1Ž2. . . . g 1Ž n.

g 2Ž2. . . . g 2Ž n.

⭈⭈⭈

⭈⭈⭈

F2 n . . . Fn n

⭈⭈⭈

g mŽ2. . . . g mŽ n.

g 1Ž1.

g 1Ž2.

⭈⭈⭈

g 1Ž n.

0

0

⭈⭈⭈

0

g 2Ž1. . . . g mŽ1.

g 2Ž2. . . . g mŽ2.

⭈⭈⭈

g 2Ž n. . . . g mŽ n.

0 . . . 0

0 . . . 0

⭈⭈⭈

0 . . . 0

⭈⭈⭈

⭈⭈⭈

,

Ž 7.53 .

292

MULTIDIMENSIONAL CALCULUS

where g jŽ i. s ⭸ g jr⭸ x i , i s 1, 2, . . . , n; j s 1, 2, . . . , m. Let ⌬ 1 denote the determinant of B 1. Furthermore, let ⌬ 2 , ⌬ 3 , . . . , ⌬ nym denote a set of principal minors of B 1 Žsee Definition 2.3.6., namely, the determinants of the principal submatrices B 2 , B 3 , . . . , B nym , where B i is obtained by deleting the first i y 1 rows and the first i y 1 columns of B 1 Ž i s 2, 3, . . . , n y m.. All the partial derivatives used in B 1 , B 2 , . . . , B nym are evaluated at x 0 . Then sufficient conditions for x 0 to be a point of local minimum of f are the following: i. If m is even, ⌬ 1 ) 0,

⌬ 2 ) 0, . . . ,

⌬ nym ) 0.

⌬ 1 - 0,

⌬ 2 - 0, . . . ,

⌬ nym - 0.

ii. If m is odd,

However, sufficient conditions for x 0 to be a point of local maximum are the following: i. If n is even, ⌬ 1 ) 0,

⌬ 2 - 0, . . . ,

Ž y1.

nym

⌬ nym - 0.

⌬ 1 - 0,

⌬ 2 ) 0, . . . ,

Ž y1.

nym

⌬ nym ) 0.

ii. If n is odd,

EXAMPLE 7.8.1. Let us find the minimum and maximum distances from the origin to the curve determined by the intersection of the plane x 2 q x 3 s 0 with the ellipsoid x 12 q 2 x 22 q x 32 q 2 x 2 x 3 s 1. Let f Ž x 1 , x 2 , x 3 . be the squared distance function from the origin, that is, f Ž x 1 , x 2 , x 3 . s x 12 q x 22 q x 32 . The equality constraints are g 1 Ž x 1 , x 2 , x 3 . ' x 2 q x 3 s 0, g 2 Ž x 1 , x 2 , x 3 . ' x 12 q 2 x 22 q x 32 q 2 x 2 x 3 y 1 s 0. Then F Ž x 1 , x 2 , x 3 . s x 12 q x 22 q x 32 q ␭1 Ž x 2 q x 3 . q ␭ 2 Ž x 12 q 2 x 22 q x 32 q 2 x 2 x 3 y 1 . ,

⭸F ⭸ x1

s 2 x 1 q 2 ␭2 x 1 s 0,

Ž 7.54 .

293

THE RIEMANN INTEGRAL OF A MULTIVARIABLE FUNCTION

⭸F ⭸ x2 ⭸F ⭸ x3

s 2 x 2 q ␭1 q 2 ␭2 Ž 2 x 2 q x 3 . s 0,

Ž 7.55 .

s 2 x 3 q ␭1 q 2 ␭2 Ž x 2 q x 3 . s 0.

Ž 7.56 .

Equations Ž7.54., Ž7.55., and Ž7.56. and the equality constraints are satisfied by the following sets of solutions: I. II. III. IV.

x 1 s 0, x 2 s 1, x 3 s y1, ␭1 s 2, ␭ 2 s y2. x 1 s 0, x 2 s y1, x 3 s 1, ␭1 s y2, ␭ 2 s y2. x 1 s 1, x 2 s 0, x 3 s 0, ␭1 s 0, ␭2 s y1. x 1 s y1, x 2 s 0, x 3 s 0, ␭1 s 0, ␭2 s y1.

To determine if any of these four sets correspond to local maxima or minima, we need to examine the values of ⌬ 1 , ⌬ 2 , . . . , ⌬ nym . Here, the matrix B 1 in formula Ž7.53. has the value 2 q 2 ␭2 0 B1 s 0 0 2 x1

0 2 q 4␭ 2 2 ␭2 1 4 x2 q 2 x3

0 2 ␭2 2 q 2 ␭2 1 2 x2 q 2 x3

0 1 1 0 0

2 x1 4 x2 q 2 x3 2 x2 q 2 x3 . 0 0

Since n s 3 and m s 2, only one ⌬ i , namely, ⌬ 1 , the determinant of B 1 , is needed. Furthermore, since m is even and n is odd, a sufficient condition for a local minimum is ⌬ 1 ) 0, and for a local maximum the condition is ⌬ 1 - 0. It can be verified that ⌬ 1 s y8 for solution sets I and II, and ⌬ 1 s 8 for solution sets III and IV. We therefore have local maxima at the points Ž0, 1,y 1. and Ž0,y 1, 1. with a common maximum value f max s 2. We also have local minima at the points Ž1, 0,0. and Žy1, 0, 0. with a common minimum value f min s 1. Since these are the only local optima on the curve of intersection, we conclude that the minimum distance from the origin to this curve is 1 and the maximum distance is '2 .

7.9. THE RIEMANN INTEGRAL OF A MULTIVARIABLE FUNCTION In Chapter 6 we discussed the Riemann integral of a real-valued function of a single variable x. In this section we extend the concept of Riemann integration to real-valued functions of n variables, x 1 , x 2 , . . . , x n .

294 Definition 7.9.1. inequalities

MULTIDIMENSIONAL CALCULUS

The set of points in R n whose coordinates satisfy the a i F x i F bi ,

i s 1, 2, . . . , n,

Ž 7.57 .

where a i - bi , i s 1, 2, . . . , n, form an n-dimensional cell denoted by c nŽ a, b .. n Ž The content Žor volume. of this cell is Ł is1 bi y a i . and is denoted by w Ž .x ␮ c n a,b . Suppose that Pi is a partition of the interval w a i , bi x, i s 1, 2, . . . , n. The n Cartesian product Ps =is1 Pi is a partition of c nŽ a, b . and consists of Ž n-dimensional subcells of c n a, b .. We denote these subcells by S1 , S2 , . . . , S␯ . The content of Si is denoted by ␮ Ž Si ., i s 1, 2, . . . , ␯ , where ␯ is the number of subcells. I We shall first define the Riemann integral of a real-valued function f Žx. on an n-dimensional cell; then we shall extend this definition to any bounded region in R n. 7.9.1. The Riemann Integral on Cells Let f : D ™ R, where D ; R n. Suppose that c nŽ a, b . is an n-dimensional cell contained in D and that f is bounded on c nŽ a, b .. Let P be a partition of c nŽ a, b . consisting of the subcells S1 , S2 , . . . , S␯ . Let m i and Mi be, respectively, the infimum and supremum of f on Si , i s 1, 2, . . . , ␯ . Consider the sums LSP Ž f . s

␯

Ý m i ␮ Ž Si . ,

Ž 7.58 .

is1

US P Ž f . s

␯

Ý Mi ␮ Ž S i . .

Ž 7.59 .

is1

We note the similarity of these sums to the ones defined in Section 6.2. As before, we refer to LSP Ž f . and US P Ž f . as the lower and upper sums, respectively, of f with respect to the partition P. The following theorem is an n-dimensional analogue of Theorem 6.2.1. The proof is left to the reader. Theorem 7.9.1. Let f : D ™ R, where D ; R n. Suppose that f is bounded on c nŽ a, b . ; D. Then f is Riemann integrable on c nŽ a, b . if and only if for every ⑀ ) 0 there exists a partition P of c nŽ a, b . such that US P Ž f . y LSP Ž f . - ⑀ . Definition 7.9.2. Let P1 and P2 be two partitions of c nŽ a, b .. Then P2 is a refinement of P1 if every point in P1 is also a point in P2 , that is, P1 ; P2 . I

THE RIEMANN INTEGRAL OF A MULTIVARIABLE FUNCTION

295

Using this definition, it is possible to prove results similar to those of Lemmas 6.2.1 and 6.2.2. In particular, we have the following lemma: Lemma 7.9.1. Let f : D ™ R, where D ; R n. Suppose that f is bounded on c nŽ a, b . ; D. Then sup P LS p Ž f . and inf P US P Ž f . exist, and sup LSP Ž f . F inf US P Ž f . . P

P

Definition 7.9.3. Let f : c nŽ a, b . ™ R be a bounded function. Then f is Riemann integrable on c nŽ a, b . if and only if sup LSP Ž f . s inf US P Ž f . . P

P

Ž 7.60 .

Their common value is called the Riemann integral of f on c nŽ a, b . and is denoted by Hc n Ž a, b. f Žx. dx. This is equivalent to the expression Hab11Hab22 ⭈⭈⭈ Hab n f Ž x 1 , x 2 , . . . , x n . dx 1 dx 2 ⭈⭈⭈ dx n . For example, for n s 2, 3 we have n

Hc Ž a, b . f Ž x. dx sHa Ha 2

b1

b2

1

2

f Ž x 1 , x 2 . dx 1 dx 2 ,

Hc Ž a, b . f Ž x. dx sHa Ha Ha 3

b1

b2

b3

1

2

3

f Ž x 1 , x 2 , x 3 . dx 1 dx 2 dx 3 .

Ž 7.61 . Ž 7.62 .

The integral in formula Ž7.61. is called a double Riemann integral, and the one in formula Ž7.62. is called a triple Riemann integral. In general, for n G 2, Hc n Ž a, b. f Žx. dx is called an n-tuple Riemann integral. I The integral Hc n Ž a, b. f Žx. dx has properties similar to those of a single-variable Riemann integral in Section 6.4. The following theorem is an extension of Theorem 6.3.1. Theorem 7.9.2. If f is continuous on an n-dimensional cell c nŽ a, b ., then it is Riemann integrable there. 7.9.2. Iterated Riemann Integrals on Cells The definition of the n-tuple Riemann integral in Section 7.9.1 does not provide a practicable way to evaluate it. We now show that the evaluation of this integral can be obtained by performing n Riemann integrals each of which is carried out with respect to one variable. Let us first consider the double integral as in formula Ž7.61.. Lemma 7.9.2. Suppose that f is real-valued and continuous on c 2 Ž a, b .. Define the function g Ž x 2 . as g Ž x2 . s

Ha

b1 1

Then g Ž x 2 . is continuous on w a2 , b 2 x.

f Ž x 1 , x 2 . dx 1 .

296

MULTIDIMENSIONAL CALCULUS

Proof. Let ⑀ ) 0 be given. Since f is continuous on c 2 Ž a, b ., which is closed and bounded, then by Theorem 7.3.2, f is uniformly continuous on c 2 Ž a, b .. We can therefore find a ␦ ) 0 such that f Ž ␰. yf Ž ␩. -

⑀ b1 y a1

if 5 ␰ y ␩ 5 - ␦ , where ␰ s Ž x 1 , x 2 .⬘, ␩ s Ž y 1 , y 2 .⬘, and x 1 , y 1 g w a1 , b1 x, x 2 , y 2 g w a2 , b 2 x. It follows that if < y 2 y x 2 < - ␦ , then g Ž y2 . y g Ž x2 . s

Ha

b1

f Ž x 1 , y 2 . y f Ž x 1 , x 2 . dx 1

1

F

Ha

b1

f Ž x 1 , y 2 . y f Ž x 1 , x 2 . dx 1

1

-

Ha

⑀

b1 1

b1 y a1

dx 1 ,

Ž 7.63 .

since 5Ž x 1 , y 2 .⬘ y Ž x 1 , x 2 .⬘ 5 s < y 2 y x 2 < - ␦ . From inequality Ž7.63. we conclude that g Ž y2 . y g Ž x2 . - ⑀ if < y 2 y x 2 < - ␦ . Hence, g Ž x 2 . is continuous on w a2 , b 2 x. Consequently, from Theorem 6.3.1, g Ž x 2 . is Riemann integrable on w a2 , b 2 x, that is, Hab22 g Ž x 2 . dx 2 exists. We call the integral

Ha

b2

g Ž x 2 . dx 2 s

2

Ha Ha b2

b1

2

1

f Ž x 1 , x 2 . dx 1 dx 2

Ž 7.64 .

I

an iterated integral of order 2.

The next theorem states that the iterated integral Ž7.64. is equal to the double integral Hc 2 Ž a, b. f Žx. dx. If f is continuous on c 2 Ž a, b ., then

Theorem 7.9.3.

Hc Ž a, b. f Ž x. dx sHa Ha 2

Proof. Exercise 7.22.

b2

b1

2

1

f Ž x 1 , x 2 . dx 1 dx 2 .

I

We note that the iterated integral in Ž7.64. was obtained by integrating first with respect to x 1 , then with respect to x 2 . This order of integration

297

THE RIEMANN INTEGRAL OF A MULTIVARIABLE FUNCTION

could have been reversed, that is, we could have integrated f with respect to x 2 and then with respect to x 1. The result would be the same in both cases. This is based on the following theorem due to Guido Fubini Ž1879᎐1943.. Theorem 7.9.4 ŽFubini’s Theorem..

Hc Ž a, b. f Ž x. dx sHa Ha 2

b2

b1

2

1

If f is continuous on c 2 Ž a, b ., then

f Ž x 1 , x 2 . dx 1 dx 2 s

Ha Ha b1

b2

1

2

Proof. See Corwin and Szczarba Ž1982, page 287..

f Ž x 1 , x 2 . dx 2 dx 1 I

A generalization of this theorem to multiple integrals of order n is given by the next theorem wsee Corwin and Szczarba Ž1982, Section 11.1.x. Theorem 7.9.5 ŽGeneralized Fubini’s Theorem.. Ž c n a, b . s
x < a i F x i F bi , i s 1, 2, . . . , n4 , then

Hc Ž a, b. f Ž x. dx sHc n

H b. a

bi

Ži. Ž ny1 a,

f Ž x . dx i dx Ž i. ,

If f is continuous on

i s 1, 2, . . . , n,

i

Ž i. Ž where dx Ž i. s dx 1 dx 2 ⭈⭈⭈ dx iy1 dx iq1 ⭈⭈⭈ dx n and c ny1 a, b . is an Ž n y 1.dimensional cell such that a1 F x 1 F b1 , a2 F x 2 F b 2 , . . . , a iy1 F x iy1 F biy1 , a iq1 F x iq1 F biq1 , . . . , a n F x n F bn .

7.9.3. Integration over General Sets We now consider n-tuple Riemann integration over regions in R n that are not necessarily cell shaped as in Section 7.9.1. Let f : D ™ R be a bounded and continuous function, where D is a bounded region in R n. There exists an n-dimensional cell c nŽ a, b . such that D ; c nŽ a, b .. Let g: c nŽ a, b . ™ R be defined as g Ž x. s

½

f Ž x. , 0,

x g D, x f D.

Then

Hc Ž a, b.g Ž x. dx sHD f Ž x. dx.

Ž 7.65 .

n

The integral on the right-hand side of Ž7.65. is independent of the choice of c nŽ a, b . provided that it contains D. It should be noted that the function g Žx. may not be continuous on Br Ž D ., the boundary of D. This, however, should not affect the existence of the integral on the left-hand side of Ž7.65.. The reason for this is given in Theorem 7.9.7. First, we need to define the so-called Jordan content of a set.

298

MULTIDIMENSIONAL CALCULUS

Definition 7.9.4. Let D ; R n be a bounded set such that D ; c nŽ a, b . for some n-dimensional cell. Let the function ␭ D : R n ™ R be defined as

␭D Ž x. s

½

x g D, x f D.

1, 0,

This is called the characteristic function of D. Suppose that sup LSP Ž ␭ D . s inf US P Ž ␭ D . , P

P

Ž 7.66 .

where LSP Ž ␭ D . and US P Ž ␭ D . are, respectively, the lower and upper sums of ␭ D Žx. with respect to a partition P of c nŽ a, b .. Then, D is said to have an n-dimensional Jordan content denoted by ␮ j Ž D ., where ␮ j Ž D . is equal to the common value of the terms in equality Ž7.66.. In this case, D is said to be Jordan measurable. I The proofs of the next two theorems can be found in Sagan Ž1974, Chapter . 11 . Theorem 7.9.6. A bounded set D ; R n is Jordan measurable if and only if its boundary Br Ž D . has a Jordan content equal to zero. Theorem 7.9.7. Let f : D ™ R, where D ; R n is bounded and Jordan measurable. If f is bounded and continuous in D except on a set that has a Jordan content equal to zero, then HD f Žx. dx exists. It follows from Theorems 7.9.6 and 7.9.7 that the integral in equality Ž7.75. must exist even though g Žx. may not be continuous on the boundary Br Ž D . of D, since Br Ž D . has a Jordan content equal to zero. EXAMPLE 7.9.1.

Let f Ž x 1 , x 2 . s x 1 x 2 and D be the region

D s
Ž x 1 , x 2 . < x 12 q x 22 F 1, x 1 G 0, x 2 G 0 4 . It is easy to see that D is contained inside the two-dimensional cell c 2 Ž 0, 1 . s
Ž x 1 , x 2 . < 0 F x 1 F 1, 0 F x 2 F 1 4 . Then

HHDx

1 x2

dx 1 dx 2 s

H0 H0Ž1yx . 1

2 1r2 1

x 1 x 2 dx 2 dx 1 .

299

THE RIEMANN INTEGRAL OF A MULTIVARIABLE FUNCTION

We note that for a fixed x 1 in w0, 1x, the part of the line through Ž x 1 , 0. that lies inside D and is parallel to the x 2-axis is in fact the interval 0 F x 2 F Ž1 y x 12 .1r2 . For this reason, the limits of x 2 are 0 and Ž1 y x 12 .1r2 . Consequently,

H0 H0Ž1yx . 1

2 1r2 1

x 1 x 2 dx 2 dx 1 s

H0 x H0Ž1yx .

s 12

1

2 1r2 1

1

x 2 dx 2 dx 1

1 H0 x Ž1 y x . dx 2 1

1

1

s 18 . In practice, it is not always necessary to make reference to c nŽ a, b . that encloses D in order to evaluate the integral on D. Rather, we only need to recognize that the limits of integration in the iterated Riemann integral depend in general on variables that have not yet been integrated out, as was seen in Example 7.9.1. Care should therefore be exercised in correctly identifying the limits of integration. By changing the order of integration Žaccording to Fubini’s theorem., it is possible to facilitate the evaluation of the integral. 2

EXAMPLE 7.9.2. Consider H HD e x 2 dx 1 dx 2 , where D is the region in the first quadrant bounded by x 2 s 1 and x 1 s x 2 . In this example, it is easier to integrate first with respect to x 1 and then with respect to x 2 . Thus

HHDe

x 22

dx 1 dx 2 s s

H0 H0 1

H0 x 1

x2

2e

2

e x 2 dx 1 dx 2

x 22

dx 2

s 12 Ž e y 1 . . EXAMPLE 7.9.3. Consider the integral H HD Ž x 12 q x 23 . dx 1 dx 2 , where D is a region in the first quadrant bounded by x 2 s x 12 and x 1 s x 24 . Hence,

HHD Ž x

2 3 1 q x2

. dx1 dx 2 s H

1

0

s

Hx'x Ž x 2

4 2

1 H0 Ž x 1 3

2 3 1 q x2

3r2 y x 212 2

. dx1

dx 2

. q ž 'x 2 y x 24 / x 23

dx 2

959 s 4680 .

7.9.4. Change of Variables in n-Tuple Riemann Integrals In this section we give an extension of the change of variables formula in Section 6.4.1 to n-tuple Riemann integrals.

300

MULTIDIMENSIONAL CALCULUS

Theorem 7.9.8. Suppose that D is a closed and bounded set in R n. Let f : D ™ R be continuous. Suppose that h: D ™ R n is a one-to-one function with continuous first-order partial derivatives such that the Jacobian determinant, det J h Ž x . s

⭸ Ž h1 , h 2 , . . . , h n .

,

⭸ Ž x1 , x 2 , . . . , x n .

is different from zero for all x in D, where x s Ž x 1 , x 2 , . . . , x n .⬘ and h i is the ith element of hŽ i s 1, 2, . . . , n.. Then

HD f Ž x. dx sHD⬘ f

g Ž u . det J g Ž u . du,

Ž 7.67 .

where D⬘ s hŽ D ., u s hŽx., g is the inverse function of h, and det J g Ž u . s

⭸ Ž g1 , g2 , . . . , gn .

Ž 7.68 .

,

⭸ Ž u1 , u 2 , . . . , u n .

where g i and u i are, respectively, the ith elements of g and u Ž i s 1, 2, . . . , n.. Proof. See, for example, Corwin and Szczarba Ž1982, Theorem 6.2., or Sagan Ž1974, Theorem 115.1.. I EXAMPLE 7.9.4. Consider the integral H HD x 1 x 22 dx 1 dx 2 , where D is bounded by the four parabolas, x 22 s x 1 , x 22 s 3 x 1 , x 12 s x 2 , x 12 s 4 x 2 . Let u1 s x 22rx 1 , u 2 s x 12rx 2 . The inverse transformation is given by x 1 s Ž u1 u 22 .

1r3

x 2 s Ž u12 u 2 .

,

1r3

.

From formula Ž7.68. we have

⭸ Ž g1 , g2 . ⭸ Ž u1 , u 2 .

s

⭸ Ž x1 , x 2 . ⭸ Ž u1 , u 2 .

sy

1 3

.

By applying formula Ž7.67. we obtain

HHDx

2 1 x2

dx 1 dx 2 s 13

HHD⬘u

5r3 4r3 1 u2

du1 du 2 ,

where D⬘ is a rectangular region in the u1 u 2 space bounded by the lines u1 s 1, 3; u 2 s 1, 4. Hence,

HHDx

2 1 x2

dx 1 dx 2 s 13

H1 u

3 5r3 1

du1

H1 u

4 4r3 2

du 2

s 563 Ž 3 8r3 y 1 . Ž 47r3 y 1 . .

301

DIFFERENTIATION UNDER THE INTEGRAL SIGN

7.10. DIFFERENTIATION UNDER THE INTEGRAL SIGN Suppose that f Ž x 1 , x 2 , . . . , x n . is a real-valued function defined on D ; R n. If some of the x i ’s, for example, x mq1 , x mq2 , . . . , x n Ž n ) m., are integrated out, we obtain a function that depends only on the remaining variables. In this section we discuss conditions under which the latter function is differentiable. For simplicity, we shall only consider functions of n s 2 variables. Theorem 7.10.1. Let f : D ™ R, where D ; R 2 contains the two-dimensional cell c 2 Ž a, b . s
Ž x 1 , x 2 .< a1 F x 1 F b1 , a2 F x 2 F b 2 4 . Suppose that f is continuous and has a continuous first-order partial derivative with respect to x 2 in D. Then, for a2 - x 2 - b 2 , d

Ha

b1

dx 2

f Ž x 1 , x 2 . dx 1 s

1

Ha

b1

⭸ f Ž x1 , x 2 . ⭸ x2

1

dx 1 .

Ž 7.69 .

Proof. Let hŽ x 2 . be defined on w a2 , b 2 x as hŽ x2 . s

Ha

b1

⭸ f Ž x1 , x 2 . ⭸ x2

1

a2 F x 2 F b 2 .

dx 1 ,

Since ⭸ fr⭸ x 2 is continuous, then by Lemma 7.9.2, hŽ x 2 . is continuous on w a2 , b 2 x. Now, let t be such that a2 - t - b 2 . By integrating hŽ x 2 . over the interval w a2 , t x we obtain

Ha h Ž x t

2

. dx 2 s H

Ha

t

2

b1

a2

1

⭸ f Ž x1 , x 2 . ⭸ x2

dx 1 dx 2 .

Ž 7.70 .

The order of integration in Ž7.70. can be reversed by Theorem 7.9.4. We than have

Ha h Ž x t

2

2

. dx 2 s H

b1

a1

s

Ha

b1

t

⭸ f Ž x1 , x 2 .

2

⭸ x2

Ha

dx 2 dx 1

f Ž x 1 , t . y f Ž x 1 , a2 . dx 1

1

s

Ha

b1

f Ž x 1 , t . dx 1 y

1

s F Ž t . y F Ž a2 . ,

Ha

b1

f Ž x 1 , a2 . dx 1

1

Ž 7.71 .

302

MULTIDIMENSIONAL CALCULUS

where F Ž y . s Hab11 f Ž x 1 , y . dx 1. If we now apply Theorem 6.4.8 and differentiate the two sides of Ž7.71. with respect to t we obtain hŽ t . s F⬘Ž t ., that is,

Ha

b1

⭸ f Ž x1 , t .

dx 1 s

⭸t

1

d

H dt a

b1

f Ž x 1 , t . dx 1 .

Ž 7.72 .

1

Formula Ž7.69. now follows from formula Ž7.72. on replacing t with x 2 .

I

Theorem 7.10.2. Let f and D be the same as in Theorem 7.10.1. Furthermore, let ␭Ž x 2 . and ␪ Ž x 2 . be functions defined and having continuous derivatives on w a2 , b 2 x such that a1 F ␭Ž x 2 . F ␪ Ž x 2 . F b1 for all x 2 in w a2 , b 2 x. Then the function G: w a2 , b 2 x ™ R defined by G Ž x2 . s

H␭␪Ž x x. f Ž x , x Ž

2.

1

2

. dx1

2

is differentiable for a2 - x 2 - b 2 , and dG dx 2

s

H␭␪Ž x x. Ž

2.

2

⭸ f Ž x1 , x 2 .

dx 1 q ␪ ⬘ Ž x 2 . f ␪ Ž x 2 . , x 2 y ␭⬘ Ž x 2 . f ␭ Ž x 2 . , x 2 .

⭸ x2

Proof. Let us write G Ž x 2 . as H Ž ␭, ␪ , x 2 .. Since both of ␭ and ␪ depend on x 2 , then by applying the total derivative formula wsee formula Ž7.12.x to H we obtain dH

s

dx 2

⭸ H d␭ ⭸␭ dx 2

q

⭸ H d␪ ⭸␪ dx 2

q

⭸H ⭸ x2

Ž 7.73 .

.

Now, by Theorem 6.4.8,

⭸H ⭸␪ ⭸H ⭸␭

s s

⭸

H fŽ x , x ⭸␪ ␭ ␪

1

⭸

H␭ f Ž x , x ␪

⭸␭

sy

2

1

⭸

2

. dx1 s f Ž ␪ , x 2 . , . dx1

H fŽ x , x ⭸␭ ␪ ␭

1

2

. dx1 s yf Ž ␭ , x 2 . .

Furthermore, by Theorem 7.10.1,

⭸H ⭸ x2

s

⭸ ⭸ x2

H␭ f Ž x , x ␪

1

. dx1 s H

␪

2

␭

⭸ f Ž x1 , x 2 . ⭸ x2

dx 1 .

303

DIFFERENTIATION UNDER THE INTEGRAL SIGN

By making the proper substitution in formula Ž7.73. we finally conclude that d dx 2

H␭␪Ž x x. f Ž x , x Ž

2.

1

2

. dx1 s H

␪ Ž x2.

2

⭸ f Ž x1 , x 2 .

dx 1 q ␪ ⬘ Ž x 2 . f ␪ Ž x 2 . , x 2

⭸ x2

␭Ž x 2 .

y ␭⬘ Ž x 2 . f ␭Ž x 2 . , x 2 .

I

EXAMPLE 7.10.1. d dx 2

Hx

cos x 2 2 2

Ž x 1 x 22 y 1 . eyx

1

dx 1 s

Hx

cos x 2

2 x 1 x 2 eyx 1 dx 1

2 2

y sin x 2 Ž x 22 cos x 2 y 1 . eycos

x2

y 2 x 2 Ž x 24 y 1 . eyx 2 . 2

Theorems 7.10.1 and 7.10.2 can be used to evaluate certain integrals of the form Hab f Ž x . dx. For example, consider the integral Is

␲

H0 x

2

cos x dx.

Define the function F Ž x2 . s

␲

H0 cos Ž x

1 x2

. dx1 ,

where x 2 G 1. Then F Ž x2 . s

1 x2

x 1 s␲

sin Ž x 1 x 2 .

s x 1 s0

1 x2

sin Ž ␲ x 2 . .

If we now differentiate F Ž x 2 . two times, we obtain F⬙ Ž x 2 . s

2 sin Ž ␲ x 2 . y 2␲ x 2 cos Ž ␲ x 2 . y ␲ 2 x 22 sin Ž ␲ x 2 . x 23

.

Thus ␲

H0 x

2 1

cos Ž x 1 x 2 . dx 1 s

2␲ x 2 cos Ž ␲ x 2 . q ␲ 2 x 22 sin Ž ␲ x 2 . y 2 sin Ž ␲ x 2 . x 23

.

304

MULTIDIMENSIONAL CALCULUS

By replacing x 2 with 1 we obtain Is

␲

H0 x

2 1

cos x 1 dx 1 s y2␲ .

7.11. APPLICATIONS IN STATISTICS Multidimensional calculus provides a theoretical framework for the study of multivariate distributions, that is, joint distributions of several random variables. It can also be used to estimate the parameters of a statistical model. We now provide details of some of these applications. Let X s Ž X 1 , X 2 , . . . , X n .⬘ be a random vector. The distribution of X is characterized by its cumulative distribution function, namely, F Ž x . s P Ž X1 F x 1 , X 2 F x 2 , . . . , X n F x n . ,

Ž 7.74 .

where x s Ž x 1 , x 2 , . . . , x n .⬘. If F Žx. is continuous and has an nth-order mixed partial derivative with respect to x 1 , x 2 , . . . , x n , then the function

⭸ nF Ž x.

f Ž x. s

⭸ x 1 ⭸ x 2 ⭈⭈⭈ ⭸ x n

,

is called the density function of X. In this case, formula Ž7.74. can be written in the form F Ž x. s

Hy⬁Hy⬁ ⭈⭈⭈ Hy⬁ f Ž z. dz. x1

x2

xn

where z s Ž z1 , z 2 , . . . , z n .⬘. If the random variable X i Ž i s 1, 2, . . . , n. is considered separately, then its distribution function is called the ith marginal distribution of X. Its density function f i Ž x i ., called the ith marginal density function, can be obtained by integrating out the remaining n y 1 variables from f Žx.. For example, if X s Ž X 1 , X 2 .⬘, then the marginal density function of X 1 is f 1Ž x1 . s

⬁

Hy⬁ f Ž x , x 1

2

. dx 2 .

Similarly, the marginal density function of X 2 is f2 Ž x2 . s

⬁

Hy⬁ f Ž x , x 1

2

. dx1 .

305

APPLICATIONS IN STATISTICS

In particular, if X 1 , X 2 , . . . , X n are independent random variables, then the density function of X s Ž X 1 , X 2 , . . . , X n .⬘ is the product of all the associated n marginal density functions, that is, f Žx. s Ł is1 f i Ž x i .. If only n y 2 variables are integrated out from f Žx., we obtain the so-called bivariate density function of the remaining two variables. For example, if X s Ž X 1 , X 2 , X 3 , X 4 .⬘, the bivariate density function of x 1 and x 2 is f 12 Ž x 1 , x 2 . s

⬁

⬁

Hy⬁Hy⬁ f Ž x , x 1

2,

x 3 , x 4 . dx 3 dx 4 .

Now, the mean of X s Ž X 1 , X 2 , . . . , X n .⬘ is ␮ s Ž ␮ 1 , ␮ 2 , . . . , ␮ n .⬘, where

␮i s

⬁

Hy⬁ x f Ž x . dx , i i

i

i s 1, 2, . . . , n.

i

The variance᎐covariance matrix of X is the n = n matrix ⌺ s Ž ␴i j ., where

␴i j s

⬁

⬁

Hy⬁Hy⬁ Ž x y ␮ . Ž x y ␮ . f i

i

j

j

ij

Ž x i , x j . dx i dx j ,

where ␮ i and ␮ j are the means of X i and X j , respectively, and f i j Ž x i , x j . is the bivariate density function of X i and X j , i / j. If i s j, then ␴i i is the variance of X i , where

␴ii s

⬁

Hy⬁ Ž x y ␮ . i

i

2

f i Ž x i . dx i ,

i s 1, 2, . . . , n.

7.11.1. Transformations of Random Vectors In this section we consider a multivariate extension of formula Ž6.73. regarding the density function of a function of a single random variable. This is given in the next theorem. Theorem 7.11.1. Let X be a random vector with a continuous density function f Žx.. Let g: D ™ R n, where D is an open subset of R n such that P ŽX g D . s 1. Suppose that g satisfies the conditions of the inverse function theorem ŽTheorem 7.6.1., namely the following: i. g has continuous first-order partial derivatives in D. ii. The Jacobian matrix J g Žx. is nonsingular in D, that is, det J g Ž x . s

⭸ Ž g1 , g2 , . . . , gn . ⭸ Ž x1 , x 2 , . . . , x n .

/0

for all x g D, where g i is the ith element of g Ž i s 1, 2, . . . , n..

306

MULTIDIMENSIONAL CALCULUS

Then the density function of Y s gŽX. is given by h Ž y . s f gy1 Ž y . det J gy1 Ž y . , where gy1 is the inverse function of g. Proof. By Theorem 7.6.1, the inverse function of g exists. Let us therefore write X s gy1 ŽY.. Now, the cumulative distribution function of Y is H Ž y. s P g 1Ž X. F y1 , s

g 2 Ž X. F y 2 , . . . ,

g n Ž X . F yn

HA f Ž x. dx,

Ž 7.75 .

n

where A n s
x g D < g i Žx. F yi , i s 1, 2, . . . , n4 . If we make the change of variable w s gŽx. in formula Ž7.75., then, by applying Theorem 7.9.8 with gy1 Žw. used instead of gŽu., we obtain

HA f Ž x. dx sHB f n

gy1 Ž w . det J gy1 Ž w . dw,

n

where Bn s gŽ A n . s
gŽx.< g i Žx. F yi , i s 1, 2, . . . , n4 . Thus H Ž y. s

Hy⬁Hy⬁ ⭈⭈⭈ Hy⬁ f y1

y2

yn

gy1 Ž w . det J gy1 Ž w .

dw.

It follows that the density function of Y is h Ž y . s f gy1 Ž y .

y1 y1 ⭸ Ž gy1 1 , g2 , . . . , gn .

⭸ Ž y 1 , y 2 , . . . , yn .

where gy1 is the ith element of gy1 Ž i s 1, 2, . . . , n.. i

Ž 7.76 .

, I

EXAMPLE 7.11.1. Let X s Ž X 1 , X 2 .⬘, where X 1 and X 2 are independent random variables that have the standard normal distribution. Here, the density function of X is the product of the density functions of X 1 and X 2 . Thus f Ž x. s

1 2␲

exp y

1 2

Ž x 12 q x 22 .

,

Let Y s Ž Y1 , Y2 .⬘ be defined as Y1 s X 1 q X 2 , Y2 s X 1 y 2 X 2 .

y⬁ - x 1 , x 2 - ⬁.

307

APPLICATIONS IN STATISTICS

In this case, the set D in Theorem 7.11.1 is R 2 , g 1Žx. s x 1 q x 2 , g 2 Žx. s x 1 y Žy. s x 1 s 13 Ž2 y 1 q y 2 ., gy1 Žy. s x 2 s 13 Ž y 1 y y 2 ., and 2 x 2 , gy1 1 2 y1 ⭸ Ž gy1 1 , g2 .

s det

⭸ Ž y1 , y 2 .

2 3

1 3

y

1 3

sy

1 3

1 3

.

Hence, by formula Ž7.76., the density function of y is h Ž y. s s

1 2␲ 1 6␲

exp y exp y

ž

/ ž

1 2 y1 q y 2 2 1 18

2

y

3

1

y1 y y 2

2

3

Ž 5 y12 q 2 y1 y 2 q 2 y 22 .

,

/

2

=

1 3

y⬁ - y 1 , y 2 - ⬁.

EXAMPLE 7.11.2. Suppose that it is desired to determine the density function of the random variable V s X 1 q X 2 , where X 1 G 0, X 2 G 0, and X s Ž X 1 , X 2 .⬘ has a continuous density function f Ž x 1 , x 2 .. This can be accomplished in two ways: i. Let QŽ ®. denote the cumulative distribution function of V and let q Ž ®. be its density function. Then Q Ž ® . s P Ž X1 q X 2 F ® . s

HHA f Ž x , x 1

2

. dx1 dx 2 ,

where A s
Ž x 1 , x 2 .< x 1 G 0, x 2 G 0, x 1 q x 2 F ®4 . We can write QŽ ®. as QŽ ®. s

®

®yx 2

H0 H0

f Ž x 1 , x 2 . dx 1 dx 2 .

If we now apply Theorem 7.10.2, we obtain q Ž ®. s s

dQ d®

s

⭸

®

H0

⭸®

®

H0 f Ž ®y x

2,

®yx 2

H0

f Ž x 1 , x 2 . dx 1 dx 2

x 2 . dx 2 .

ii. Consider the following transformation: Y1 s X 1 q X 2 , Y2 s X 2 .

Ž 7.77 .

308

MULTIDIMENSIONAL CALCULUS

Then X 1 s Y1 y Y2 , X 2 s Y2 . By Theorem 7.11.1, the density function of Y s Ž Y1 , Y2 .⬘ is h Ž y1 , y 2 . s f Ž y1 y y 2 , y 2 .

⭸ Ž x1 , x 2 . ⭸ Ž y1 , y 2 .

s f Ž y 1 y y 2 , y 2 . det

1 0

y1 1

s f Ž y 1 y y 2 , y 2 . , y 1 G y 2 G 0. By integrating y 2 out we obtain the marginal density function of Y1 s V, namely, q Ž ®. s

®

H0 f Ž y y y 1

2,

y 2 . dy 2 s

®

H0 f Ž ®y x

2,

x 2 . dx 2 .

This is identical to the density function given in formula Ž7.77.. 7.11.2. Maximum Likelihood Estimation Let X 1 , X 2 , . . . , X n be a sample of size n from a population whose distribution depends on a set of p parameters, namely ␪ 1 , ␪ 2 , . . . , ␪p . We can regard this sample as forming a random vector X s Ž X 1 , X 2 , . . . , X n .⬘. Suppose that X has the density function f Žx, ␪ ., where x s Ž x 1 , x 2 , . . . , x n .⬘ and ␪ s Ž ␪ 1 , ␪ 2 , . . . , ␪p .⬘. This density function is usually referred to as the likelihood function of X; we denote it by LŽx, ␪ .. For a given sample, the maximum likelihood estimate of ␪, denoted by ˆ ␪, is the value of ␪ that maximizes LŽx, ␪ .. If LŽx, ␪ . has partial derivatives with respect to ␪ 1 , ␪ 2 , . . . , ␪p , then ˆ ␪ is often obtained by solving the equations

⭸ L Ž x, ˆ ␪. ⭸␪ i

s 0,

i s 1, 2, . . . , p.

In most situations, it is more convenient to work with the natural logarithm of LŽx, ␪ .; its maxima are attained at the same points as those of LŽx, ␪ .. Thus ˆ ␪ satisfies the equation

⭸ log L Ž x, ˆ ␪. ⭸␪ i

s 0,

i s 1, 2, . . . , p.

Equations Ž7.78. are known as the likelihood equations.

Ž 7.78 .

309

APPLICATIONS IN STATISTICS

EXAMPLE 7.11.3. Suppose that X 1 , X 2 , . . . , X n form a sample of size n from a normal distribution with an unknown mean ␮ and a variance ␴ 2 . Here, ␪ s Ž ␮ , ␴ 2 .⬘, and the likelihood function is given by L Ž x, ␪ . s

1

Ž 2␲␴ 2 .

nr2

exp y

n

1 2␴

Ý Ž xi y ␮. 2

2

,

is1

Let L*Žx, ␪ . s log LŽx, ␪ .. Then L* Ž x, ␪ . s y

n

1 2␴

n

Ý Ž x i y ␮ . 2 y 2 log Ž 2␲␴ 2 . .

2

is1

The likelihood equations in formula Ž7.78. are of the form

⭸ L* ⭸␮ ⭸ L* ⭸␴

2

s

s

n

1

␴

2

Ý Ž x i y ␮ˆ . s 0

Ž 7.79 .

is1

n

1

Ý Ž x y ␮ˆ . 2 ␴ˆ is1 i

2

4

y

n 2 ␴ˆ 2

s 0.

Ž 7.80 .

Equations Ž7.79. and Ž7.80. can be written as nŽ x y ␮ ˆ . s 0, n

Ý Ž x i y ␮ˆ .

2

Ž 7.81 . Ž 7.82 .

y n ␴ˆ 2 s 0,

is1

where xs Ž1rn.Ý nis1 x i . If n G 2, then equations Ž7.81. and Ž7.82. have the solution

␮ ˆ s x, ␴ˆ 2 s

1 n

n

Ý Ž xi y x .

2

.

is1

These are the maximum likelihood estimates of ␮ and ␴ 2 , respectively. It can be verified that ␮ ˆ and ␴ˆ 2 are indeed the values of ␮ and ␴ 2 that maximize L*Žx, ␪ .. To show this, let us consider the Hessian matrix A of second-order partial derivatives of L* Žsee formula 7.34.,

As

⭸ 2 L*

⭸ 2 L*

⭸␮2

⭸␮ ⭸␴ 2

⭸ 2 L*

⭸ 2 L*

⭸␮ ⭸␴ 2

⭸␴ 4

.

310

MULTIDIMENSIONAL CALCULUS

Hence, for ␮ s ␮ ˆ and ␴ 2 s ␴ˆ 2 ,

⭸ 2 L* ⭸␮

2

⭸ 2 L* ⭸␮ ⭸␴ 2 ⭸ 2 L* ⭸␴

4

sy

sy

sy

n

␴ˆ 2

, n

1

␴ˆ 4

Ý Ž x i y ␮ˆ . s 0,

is1

n 2 ␴ˆ 4

.

Thus ⭸ 2 L*r⭸␮2 - 0 and detŽA. s n2r2 ␴ˆ 6 ) 0. Therefore, by Corollary 7.7.1, Ž␮ ˆ , ␴ˆ 2 . is a point of local maximum of L*. Since it is the only maximum, it must also be the absolute maximum. Maximum likelihood estimators have interesting asymptotic properties. For more information on these properties, see, for example, Bickel and Doksum Ž1977, Section 4.4.. 7.11.3. Comparison of Two Unbiased Estimators Let X 1 and X 2 be two unbiased estimators of a parameter ␮. Suppose that X s Ž X 1 , X 2 .⬘ has the density function f Ž x 1 , x 2 ., y⬁ - x 1 , x 2 - ⬁. To compare these estimators, we may consider the probability that one estimator, for example, X 1 , is closer to ␮ than the other, X 2 , that is, ps P < X 1 y ␮ < - < X 2 y ␮ < . This probability can be expressed as ps

HHD f Ž x , x 1

2

. dx1 dx 2 ,

Ž 7.83 .

where D s
Ž x 1 , x 2 .< < x 1 y ␮ < - < x 2 y ␮ < 4 . Let us now make the following change of variables using polar coordinates: x 1 y ␮ s r cos ␪ ,

x 2 y ␮ s r sin ␪ .

By applying formula Ž7.67., the integral in Ž7.83. can be written as ps s

HHD⬘˜g Ž r , ␪ .

⭸ Ž x1 , x 2 . ⭸ Ž r, ␪ .

HHD⬘˜g Ž r , ␪ . r dr d␪ ,

dr d␪

311

APPLICATIONS IN STATISTICS

where ˜ g Ž r, ␪ . s f Ž ␮ q r cos ␪ , ␮ q r sin ␪ . and

½

D⬘ Ž r , ␪ . < 0 F r - ⬁,

␲ 4

5

3␲ 5␲ 7␲ , F␪F . 4 4 4

F␪F

In particular, if X has the bivariate normal density, then f Ž x1 , x 2 . s

1 2␲␴ 1 ␴ 2 Ž 1 y ␳ 2 .

½

= exp y

1r2

Ž x1 y ␮ .

1 2 Ž1 y ␳

2

.

␴ 12 q

2

y

2 ␳ Ž x1 y ␮ . Ž x 2 y ␮ .

Ž x2 y ␮ .

␴1 ␴ 2 2

␴ 22

5

,

y⬁ - x 1 , x 2 - ⬁,

and

˜g Ž r , ␪ . s

1 2␲␴ 1 ␴ 2 Ž 1 y ␳ 2 .

½

= exp y

1r2

r2

cos 2 ␪

2 Ž1 y ␳ 2 .

␴ 12

y

2 ␳ cos ␪ sin ␪

␴1 ␴ 2

q

sin 2 ␪

␴ 22

5

,

where ␴ 12 and ␴ 22 are the variances of X 1 and X 2 , respectively, and ␳ is their correlation coefficient. In this case, ⬁

H␲r4 H0

ps 2

3␲r4

˜g Ž r , ␪ . r dr d␪ .

It can be shown Žsee Lowerre, 1983. that ps 1 y

1

␲

Arctan

2 ␴1 ␴ 2 Ž1 y ␳ 2 .

␴ 22 y ␴ 12

1r2

Ž 7.84 .

if ␴ 2 ) ␴ 1. A large value of p indicates that X 1 is closer to ␮ than X 2 , which means that X 1 is a better estimator of ␮ than X 2 . 7.11.4. Best Linear Unbiased Estimation Let X 1 , X 2 , . . . , X n be independent and identically distributed random variables with a common mean ␮ and a common variance ␴ 2 . An estimator of

312

MULTIDIMENSIONAL CALCULUS

the form ␾ˆs Ý nis1 a i X i , where the a i ’s are constants, is said to be a linear estimator of ␮. This estimator is unbiased if E Ž ␾ˆ. s ␮ , that is, if Ý nis1 a i s 1, since E Ž X i . s ␮ for i s 1, 2, . . . , n. The variance of ␾ˆ is given by Var Ž ␾ˆ . s ␴ 2

n

Ý a2i .

is1

The smaller the variance of ␾ˆ, the more efficient ␾ˆ is as an estimator of ␮. In particular, if a1 , a2 , . . . , a n are chosen so that VarŽ ␾ˆ. attains a minimum value, then ␾ˆ will have the smallest variance among all unbiased linear estimators of ␮. In this case, ␾ˆ is called the best linear unbiased estimator ŽBLUE. of ␮. Thus to find the BLUE of ␮ we need to minimize the function f s Ý nis1 a2i subject to the constraint Ý nis1 a i s 1. This minimization problem can be solved using the method of Lagrange multipliers. Let us therefore write F wsee formula Ž7.41.x as n

Fs

Ý

is1

⭸F ⭸ ai

žÝ / n

a2i q ␭

ai y 1 ,

is1

s 2 a i q ␭ s 0,

i s 1, 2, . . . , n.

Hence, a i s y␭r2 Ž i s 1, 2, . . . , n.. Using the constraint Ý nis1 a i s 1, we conclude that ␭ s y2rn. Thus a i s 1rn, i s 1, 2, . . . , n. To verify that this solution minimizes f, we need to consider the signs of ⌬ 1 , ⌬ 2 , . . . , ⌬ ny1 , where ⌬ i is the determinant of B i Žsee Section 7.8.. Here, B 1 is an Ž n q 1. = Ž n q 1. matrix of the form B1 s

2I n 1Xn

1n . 0

It follows that ⌬ 1 s det Ž B 1 . s y ⌬2 sy

n2 n 2

Ž n y 1 . 2 ny1 2

- 0,

- 0,

. . . ⌬ ny1 s y2 2 - 0. Since the number of constraints, m s 1, is odd, then by the sufficient

313

APPLICATIONS IN STATISTICS

conditions described in Section 7.8 we must have a local minimum when a i s 1rn, i s 1, 2, . . . , n. Since this is the only local minimum in R n, it must be the absolute minimum. Note that for such values of a1 , a2 , . . . , a n , ␾ˆ is the sample mean X n . We conclude that the sample mean is the most efficient Žin terms of variance. unbiased linear estimator of ␮. 7.11.5. Optimal Choice of Sample Sizes in Stratified Sampling In stratified sampling, a finite population of N units is divided into r subpopulations, called strata, of sizes N1 , N2 , . . . , Nr . From each stratum a random sample is drawn, and the drawn samples are obtained independently in the different strata. Let n i be the size of the sample drawn from the ith stratum Ž i s 1, 2, . . . , r .. Let yi j denote the response value obtained from the jth unit within the ith stratum Ž i s 1, 2, . . . , r; j s 1, 2, . . . , n i .. The population mean Y is Ys

Ni

r

1

r

1

Ý Ý yi j s N Ý Ni Yi ,

N

is1 js1

is1

where Yi is the true mean for the ith stratum Ž i s 1, 2, . . . , r .. A stratified estimate of Y is yst Žst for stratified ., where yst s

1 N

r

Ý Ni yi ,

is1

i in which yi s Ž1rn i .Ý njs1 yi j is the mean of the sample from the ith stratum Ž i s 1, 2, . . . , r .. If, in every stratum, yi is unbiased for Yi , then yst is an unbiased estimator of Y. The variance of yst is

Var Ž yst . s

1 N2

r

Ý Ni 2 Var Ž yi . .

is1

Since yi is the mean of a random sample from a finite population, then its variance is given by Žsee Cochran, 1963, page 22. Var Ž yi . s

Si2 ni

Ž 1 y fi . ,

i s 1, 2, . . . , r ,

where f i s n irNi , and Si2 s

1 Ni y 1

Ni

Ý ž yi j y Yi /

js1

2

.

314

MULTIDIMENSIONAL CALCULUS

Hence, Var Ž yst . s

r

1

is1

ni

Ý

L2i Si2 Ž 1 y f i . ,

where L i s NirN Ž i s 1, 2, . . . , r .. The sample sizes n1 , n 2 , . . . , n r can be chosen by the sampler in an optimal way, the optimality criterion being the minimization of VarŽ yst . for a specified cost of taking the samples. Here, the cost is defined by the formula r

cost s c 0 q

Ý ci n i ,

is1

where c i is the cost per unit in the ith stratum Ž i s 1, 2, . . . , r . and c 0 is the overhead cost. Thus the optimal choice of the sample sizes is reduced to finding the values of n1 , n 2 , . . . , n r that minimize Ý ris1 Ž1rn i . L2i Si2 Ž1 y f i . subject to the constraint r

Ý c i n i s dy c0 ,

Ž 7.85 .

is1

where d is a constant. Using the method of Lagrange multipliers, we write Fs

s

r

1

is1

ni

r

1

Ý

Ý

is1

ni

L2i Si2 Ž 1 y f i . q ␭ r

L2i Si2 y

Ý

is1

1 Ni

ž

r

Ý c i n i q c0 y d

is1

žÝ

/

r

L2i Si2 q ␭

Differentiating with respect to n i Ž i s 1, 2, . . . , r ., we obtain

⭸F ⭸ ni

sy

1 n2i

L2i Si2 q ␭ c i s 0,

i s 1, 2, . . . , r ,

Thus n i s Ž ␭ci .

y1r2

i s 1, 2, . . . , r .

L i Si ,

By substituting n i in the equality constraint Ž7.85. we get

'␭ s

'

Ý ris1 c i L i Si dy c 0

.

/

c i n i q c0 y d .

is1

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

315

Therefore,

ni s

Ž dy c0 . Ni Si

'c Ý 'c N S i

r js1

j

j

,

i s 1, 2, . . . , r .

Ž 7.86 .

j

It is easy to verify Žusing the sufficient conditions in Section 7.8. that the values of n1 , n 2 , . . . , n r given by equation Ž7.86. minimize VarŽ yst . under the constraint of equality Ž7.85.. We conclude that VarŽ yst . is minimized when n i is proportional to Ž1r c i . Ni S i Ž i s 1, 2, . . . , r .. Consequently, n i must be large if the corresponding stratum is large, if the cost of sampling per unit in that stratum is low, or if the variability within the stratum is large.

'

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

Bickel, P. J., and K. A. Doksum Ž1977.. Mathematical Statistics, Holden-Day, San Francisco. ŽChap. 1 discusses distribution theory for transformation of random vectors.. Brownlee, K. A. Ž1965.. Statistical Theory and Methodology, 2nd ed. Wiley, New York. ŽSee Section 9.8 with regard to the Behrens᎐Fisher test.. Cochran, W. G. Ž1963.. Sampling Techniques, 2nd ed. Wiley, New York. ŽThis is a classic book on sampling theory as developed for use in sample surveys.. Corwin, L. J., and R. H. Szczarba Ž1982.. Multi®ariate Calculus. Marcel Dekker, New York. ŽThis is a useful book that provides an introduction to multivariable calculus. The topics covered include continuity, differentiation, multiple integrals, line and surface integrals, differential forms, and infinite series. . Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. ŽChap. 8 discusses limits and continuity for a multivariable function; Chap. 10 covers the inverse function theorem; Chap. 11 discusses multiple integration. . Gillespie, R. P. Ž1954.. Partial Differentiation. Oliver and Boyd, Edinburgh, Scotland. ŽThis concise book provides a brief introduction to multivariable calculus. It covers partial differentiation, Taylor’s theorem, and maxima and minima of functions of several variables. . Kaplan, W. Ž1991.. Ad®anced Calculus, 4th ed. Addison-Wesley, Redwood City, California. ŽTopics pertaining to multivariable calculus are treated in several chapters including Chaps. 2, 3, 4, 5, and 6.. Kaplan, W., and D. J. Lewis Ž1971.. Calculus and Linear Algebra, Vol. II. Wiley, New York. ŽChap. 12 gives a brief introduction to differential calculus of a multivariable function; Chap. 13 covers multiple integration. . Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. ŽMultivariate transformations are discussed in Chap. 10.. Lowerre, J. M. Ž1983.. ‘‘An integral of the bivariate normal and an application.’’ Amer. Statist., 37, 235᎐236. Rudin, W. Ž1964.. Principles of Mathematical Analysis, 2nd ed. McGraw-Hill, New York. ŽChap. 9 includes a study of multivariable functions. .

316

MULTIDIMENSIONAL CALCULUS

Sagan, H. Ž1974.. Ad®anced Calculus. Houghton Mifflin, Boston. ŽChap. 9 covers differential calculus of a multivariable function; Chap. 10 deals with the inverse function and implicit function theorems; Chap. 11 discusses multiple integration. . Satterthwaite, F. E. Ž1946.. ‘‘An approximate distribution of estimates of variance components.’’ Biometrics Bull., 2, 110᎐114. Taylor, A. E., and W. R. Mann Ž1972.. Ad®anced Calculus, 2nd ed. Wiley, New York. ŽThis book contains several chapters on multivariable calculus with many helpful exercises. . Thibaudeau, Y., and G. P. H. Styan Ž1985.. ‘‘Bounds for Chakrabarti’s measure of imbalance in experimental design.’’ In Proceedings of the First International Tampere Seminar on Linear Statistical Models and Their Applications, T. Pukkila and S. Puntanen, eds. University of Tampere, Tampere, Finland, pp. 323᎐347. Wen, L. Ž2001.. ‘‘A counterexample for the two-dimensional density function.’’ Amer. Math. Monthly, 108, 367᎐368.

EXERCISES In Mathematics 7.1. Let f Ž x 1 , x 2 . be a function defined on R 2 as

f Ž x1 , x 2

° < x < exp y < x < x x ¢0,

. s~

ž

1 2 2

1 2 2

/

,

x 2 / 0, x 2 s 0.

(a) Show that f Ž x 1 , x 2 . has a limit equal to zero as x s Ž x 1 , x 2 .⬘ ™ 0 along any straight line through the origin. (b) Show that f Ž x 1 , x 2 . does not have a limit as x ™ 0. 7.2. Prove Lemma 7.3.1. 7.3. Prove Lemma 7.3.2. 7.4. Prove Lemma 7.3.3. 7.5. Consider the function

°xx f Ž x , x . s~ x q x ¢0, 1

1

2

2 1

2

2 2

,

Ž x 1 , x 2 . / Ž 0, 0 . , Ž x 1 , x 2 . s Ž 0, 0 . .

(a) Show that f Ž x 1 , x 2 . is not continuous at the origin.

317

EXERCISES

(b) Show that the partial derivatives of f Ž x 1 , x 2 . with respect to x 1 and x 2 exist at the origin. w Note: This exercise shows that a multivariable function does not have to be continuous at a point in order for its partial derivatives to exist at that point.x 7.6. The function f Ž x 1 , x 2 , . . . , x k . is said to be homogeneous of degree n in x 1 , x 2 , . . . , x k if for any nonzero scalar t, f Ž tx 1 , tx 2 , . . . , tx k . s t n f Ž x 1 , x 2 , . . . , x k . for all x s Ž x 1 , x 2 , . . . , x k .⬘ in the domain of f Ž x 1 , x 2 , . . . , x k . is homogeneous of degree n, then

⭸f

k

Ý xi ⭸ x

is1

f. Show that if

s nf i

w Note: This result is known as Euler’s theorem for homogeneous functions. x 7.7. Consider the function

°xx f Ž x , x . s~ x q x ¢0, 2 1 2

1

2

4 1

2 2

,

Ž x 1 , x 2 . / Ž 0, 0 . , Ž x 1 , x 2 . s Ž 0, 0 . .

(a) Is f continuous at the origin? Why or why not? (b) Show that f has a directional derivative in every direction at the origin. 7.8. Let S be a surface defined by the equation f Žx. s c 0 , where x s Ž x 1 , x 2 , . . . , x k .⬘ and c 0 is a constant. Let C denote a curve on S given by the equations x 1 s g 1Ž t ., x 2 s g 2 Ž t ., . . . , x k s g k Ž t ., where g 1 , g 2 , . . . , g k are differentiable functions. Let s be the arc length of C measured from some fixed point in such a way that s increases with t. The curve can then be parameterized, using s instead of t, in the form x 1 s h1Ž s ., x 2 s h 2 Ž s ., . . . , x k s h k Ž s .. Suppose that f has partial derivatives with respect to x 1 , x 2 , . . . , x k . Show that the directional derivative of f at a point x on C in the direction of v, where v is a unit tangent vector to C at x Žin the direction of increasing s ., is equal to dfrds. 7.9. Use Taylor’s expansion in a neighborhood of the origin to obtain a second-order approximation for each of the following functions:

318

MULTIDIMENSIONAL CALCULUS

(a) f Ž x 1 , x 2 . s expŽ x 2 sin x 1 .. (b) f Ž x 1 , x 2 , x 3 . s sinŽ e x 1 q x 22 q x 33 .. (c) f Ž x 1 , x 2 . s cosŽ x 1 x 2 .. 7.10. Suppose that f Ž x 1 , x 2 . and g Ž x 1 , x 2 . are continuously differentiable functions in a neighborhood of a point x 0 s Ž x 10 , x 20 .⬘. Consider the equation u1 s f Ž x 1 , x 2 .. Suppose that ⭸ fr⭸ x 1 / 0 at x 0 . (a) Show that

⭸ x1 ⭸ x2

sy

⭸f

⭸f

⭸ x2

⭸ x1

in a neighborhood of x 0 . (b) Suppose that in a neighborhood of x 0 ,

⭸Ž f, g.

s 0.

⭸ Ž x1 , x 2 . Show that

⭸ g ⭸ x1 ⭸ x1 ⭸ x 2

q

⭸g ⭸ x2

s 0,

that is, g is actually independent of x 2 in a neighborhood of x 0 . (c) Deduce from Žb. that there exists a function ␾ : D ™ R, where D ; R is a neighborhood of f Žx 0 ., such that g Ž x1 , x 2 . s ␾ f Ž x1 , x 2 . throughout a neighborhood of x 0 . In this case, the functions f and g are said to be functionally dependent. (d) Show that if f and g are functionally dependent, then

⭸Ž f, g. ⭸ Ž x1 , x 2 .

s 0.

w Note: From Žb., Žc., and Žd. we conclude that f and g are functionally dependent on a set ⌬ ; R 2 if and only if ⭸ Ž f, g .r⭸ Ž x 1 , x 2 . s 0 in ⌬.x 7.11. Consider the equation x1

⭸u ⭸ x1

q x2

⭸u ⭸ x2

q x3

⭸u ⭸ x3

s nu.

319

EXERCISES

Let ␰ 1 s x 1rx 3 , ␰ 2 s x 2rx 3 , ␰ 3 s x 3 . Use this change of variables to show that the equation can be written as

␰3

⭸u ⭸␰ 3

s nu.

Deduce that u is of the form u s x 3n F

ž

/

x1 x 2 , . x3 x3

7.12. Let u1 and u 2 be defined as u1 s x 1 Ž 1 y x 22 . u 2 s Ž 1 y x 12 .

1r2

1r2

q x 2 Ž 1 y x 12 .

Ž 1 y x 22 .

1r2

1r2

,

y x1 x 2 .

Show that u1 and u 2 are functionally dependent. 7.13. Let f: R 3 ™ R 3 be defined as u s f Ž x. ,

x s Ž x 1 , x 2 , x 3 . ⬘,

u s Ž u1 , u 2 , u 3 . ⬘,

where u1 s x 13, u 2 s x 23 , u 3 s x 33. (a) Show that the Jacobian matrix of f is not nonsingular in any subset D ; R 3 that contains points on ay of the coordinate planes. (b) Show that f has a unique inverse everywhere in R 3 including any subset D of the type described in Ža.. w Note: This exercise shows that the nonvanishing of the Jacobian determinant in Theorem 7.6.1 Žinverse function theorem. is a sufficient condition for the existence of an inverse function, but is not necessary. x 7.14. Consider the equations g 1 Ž x 1 , x 2 , y 1 , y 2 . s 0, g 2 Ž x 1 , x 2 , y 1 , y 2 . s 0, where g 1 and g 2 are differentiable functions defined on a set D ; R 4 . Suppose that ⭸ Ž g 1 , g 2 .r⭸ Ž x 1 , x 2 . / 0 in D. Show that

⭸ x1 ⭸ y1 ⭸ x2 ⭸ y1

sy

sy

⭸ Ž g1 , g2 .

⭸ Ž g1 , g2 .

⭸ Ž y1 , x 2 .

⭸ Ž x1 , x 2 .

⭸ Ž g1 , g2 .

⭸ Ž g1 , g2 .

⭸ Ž x1 , y1 .

⭸ Ž x1 , x 2 .

,

.

320

MULTIDIMENSIONAL CALCULUS

7.15. Let f Ž x 1 , x 2 , x 3 . s 0, g Ž x 1 , x 2 , x 3 . s 0, where f and g are differentiable functions defined on a set D ; R 3. Suppose that

⭸Ž f, g. ⭸ Ž x2 , x3 .

⭸Ž f, g.

/ 0,

⭸ Ž x 3 , x1 .

⭸Ž f, g.

/ 0,

⭸ Ž x1 , x 2 .

/0

in D. Show that dx 1

⭸ Ž f , g . r⭸ Ž x 2 , x 3 .

s

dx 2

⭸ Ž f , g . r⭸ Ž x 3 , x 1 .

s

dx 3

⭸ Ž f , g . r⭸ Ž x 1 , x 2 .

.

7.16. Determine the stationary points of the following functions and check for local minima and maxima: (a) f s x 12 q x 22 q x 1 q x 2 q x 1 x 2 . (b) f s 2 ␣ x 12 y x 1 x 2 q x 22 q x 1 y x 2 q 1, where ␣ is a scalar. Can ␣ be chosen so that the stationary point is Ži. a point of local minimum; Žii. a point of local maximum; Žiii. a saddle point? (c) f s x 13 y 6 x 1 x 2 q 3 x 22 y 24 x 1 q 4. (d) f s x 14 q x 24 y 2Ž x 1 y x 2 . 2 . 7.17. Consider the function fs

1 q pq Ý m is1 pi

Ž 1 q p 2 q Ýmis1 pi2 .

1r2

,

which is defined on the region C s
Ž p1 , p 2 , . . . , pm . < 0 - pF pi F 1, i s 1, 2, . . . , m4 , where p is a known constant. Show that (a) ⭸ fr⭸ pi , for i s 1, 2, . . . , m, vanish at exactly one point in C. (b) The gradient vector ⵱f s Ž ⭸ fr⭸ p1 , ⭸ fr⭸ p 2 , . . . , ⭸ fr⭸ pm .⬘ does not vanish anywhere on the boundary of C. (c) f attains its absolute maximum in the interior of C at the point Ž p1o , p 2o , . . . , pmo ., where pio s

1 q p2 1qp

,

i s 1, 2, . . . , m.

w Note: The function f was considered in an article by Thibaudeau and Styan Ž1985. concerning a measure of imbalance for experimental designs.x 7.18. Show that the function f s Ž x 2 y x 12 .Ž x 2 y 2 x 12 . does not have a local maximum or minimum at the origin, although it has a local minimum for t s 0 along every straight line given by the equations x 1 s at, x 2 s bt, where a and b are constants.

321

EXERCISES

7.19. Find the optimal values of the function f s x 12 q 12 x 1 x 2 q 2 x 22 subject to 4 x 12 q x 22 s 25. Determine the nature of the optima. 7.20. Find the minimum distance from the origin to the curve of intersection of the surfaces, x 3 Ž x 1 q x 2 . s y2 and x 1 x 2 s 1. 7.21. Apply the method of Lagrange multipliers to show that

Ž x 12 x 22 x 32 .

1r3

F 13 Ž x 12 q x 22 q x 32 .

for all values of x 1 , x 2 , x 3 . w Hint: Find the maximum value of f s x 12 x 22 x 32 subject to x 12 q x 22 q x 32 s c 2 , where c is a constant. x 7.22. Prove Theorem 7.9.3. 7.23. Evaluate the following integrals: (a) H HD x 2 x 1 dx 1 dx 2 , where

'

D s
Ž x 1 , x 2 . < x 1 ) 0, x 2 ) x 12 , x 2 - 2 y x 12 4 . (b) H01 w H0 '1yx 2 Ž 23 x 1 q 43 x 2 . dx 1 x dx 2 . 2

7.24. Show that if f Ž x 1 , x 2 . is continuous, then

H0 Hx4 x yx f Ž x , x 2

1

2 1

1

. dx 2 dx1 s H

4

2 1

2

0

H2y'x'4yx

f Ž x 1 , x 2 . dx 1 dx 2 .

2

2

7.25. Consider the integral Is

1yx fŽ x , x H0 H1yx 1

2 1

1

2

. dx 2 dx1 .

1

(a) Write an equivalent expression for I by reversing the order of integration. 1yx 12 Ž (b) If g Ž x 1 . s H1yx f x 1 , x 2 . dx 2 , find dgrdx1. 1 7.26. Evaluate H HD x 1 x 2 dx 1 dx 2 , where D is a region enclosed by the four parabolas x 22 s x 1 , x 22 s 2 x 1 , x 12 s x 2 , x 12 s 2 x 2 . w Hint: Use a proper change of variables. x

322

MULTIDIMENSIONAL CALCULUS

7.27. Evaluate H H HD Ž x 12 q x 22 . dx 1 dx 2 dx 3 , where D is a sphere of radius 1 centered at the origin. w Hint: Make a change of variables using spherical polar coordinates of the form x 1 s r sin ␪ cos ␾ , x 2 s r sin ␪ sin ␾ , x 3 s r cos ␪ , 0 F r F 1, 0 F ␪ F ␲ , 0 F ␾ F 2␲ .x 7.28. Find the value of the integral Is

H0'3

dx

Ž1qx2 .

3

.

w Hint: Consider the integral H0 '3 dxrŽ aq x 2 ., where a) 0.x In Statistics 7.29. Suppose that the random vector X s Ž X 1 , X 2 .⬘ has the density function f Ž x1 , x 2 . s

½

x1 q x 2 , 0

0 - x 1 - 1, 0 - x 2 - 1, elsewhere .

(a) Are the random variables X 1 and X 2 independent? (b) Find the expected value of X 1 X 2 . 7.30. Consider the density function f Ž x 1 , x 2 . of X s Ž X 1 , X 2 .⬘, where f Ž x1 , x 2 . s

½

1, 0

yx 2 - x 1 - x 2 , 0 - x 2 - 1, elsewhere .

Show that X 1 and X 2 are uncorrelated random variables wthat is, E Ž X 1 X 2 . s E Ž X 1 . E Ž X 2 .x, but are not independent. 7.31. The density function of X s Ž X 1 , X 2 .⬘ is given by

f Ž x1 , x 2

° ¢0

1

x . s~ ⌫ Ž ␣ . ⌫ Ž ␤ .

␣y1 ␤y1 x2 1

eyx 1yx 2 ,

0 - x 1 , x 2 - ⬁, elsewhere .

323

EXERCISES

where ␣ ) 0, ␤ ) 0, and ⌫ Ž m. is the gamma function ⌫ Ž m. s H0⬁ eyx x my1 dx, m ) 0. Suppose that Y1 and Y2 are random variables defined as Y1 s

X1 X1 q X 2

,

Y2 s X 1 q X 2 . (a) Find the joint density function of Y1 and Y2 . (b) Find the marginal densities of Y1 and Y2 . (c) Are Y1 and Y2 independent? 7.32. Suppose that X s Ž X 1 , X 2 .⬘ has the density function f Ž x1 , x 2 . s

½

10 x 1 x 22 , 0

0 - x 1 - x 2 , 0 - x 2 - 1, elsewhere .

Find the density function of W s X 1 X 2 . 7.33. Find the density function of W s Ž X 12 q X 22 .1r2 given that X s Ž X 1 , X 2 .⬘ has the density function f Ž x1 , x 2 . s

½

4 x 1 x 2 eyx 1 yx 2 , 0 2

2

x 1 ) 0, x 2 ) 0, elsewhere .

7.34. Let X 1 , X 2 , . . . , X n be independent random variables that have the exponential density f Ž x . s eyx , x) 0. Let Y1 , Y2 , . . . , Yn be n random variables defined as Y1 s X 1 , Y2 s X 1 q X 2 , . . . Yn s X 1 q X 2 q ⭈⭈⭈ qX n . Find the density of Y s Ž Y1 , Y2 , . . . , Yn .⬘, and then deduce the marginal density of Yn . 7.35. Prove formula Ž7.84.. 7.36. Let X 1 and X 2 be independent random variables such that W1 s Ž6r␴ 12 . X 1 and W2 s Ž8r␴ 22 . X 2 have the chi-squared distribution with

324

MULTIDIMENSIONAL CALCULUS

six and eight degrees of freedom, respectively, where ␴ 12 and ␴ 22 are unknown parameters. Let ␪ s 17 ␴ 12 q 19 ␴ 22 . An unbiased estimator of ␪ is given by ␪ˆs 71 X 1 q 91 X 2 , since X 1 and X 2 are unbiased estimators of ␴ 12 and ␴ 22 , respectively. Using Satterthwaite’s approximation Žsee Satterthwaite, 1946., it can be shown that ␩ ␪ˆr␪ is approximately distributed as a chi-squared variate with ␩ degrees of freedom, where ␩ is given by

Ž 17 ␴12 q 19 ␴ 22 . , 1 1 1 1 2 2 2 2 6 Ž 7 ␴1 . q 8 Ž 9 ␴2 . 2

␩s which can be written as

␩s

8 Ž 9 q 7␭ .

2

108 q 49␭2

,

where ␭ s ␴ 22r␴ 12 . It follows that the probability ps P

ž

␩ ␪ˆ 2 ␹ 0.025 ,␩

␩ ␪ˆ

-␪-

2 ␹ 0.975 ,␩

/

,

where ␹␣2, ␩ denotes the upper 100 ␣ % point of the chi-squared distribution with ␩ degrees of freedom, is approximately equal to 0.95. Compute the exact value of p using double integration, given that ␭ s 2. Compare the result with the 0.95 value. w Notes: Ž1. The density function of a chi-squared random variable with n degrees of freedom is given in Example 6.9.6. Ž2. In general, ␩ is unknown. It can be estimated by ␩ ˆ which results from replacing ␭ with ˆ s X 2rX1 in the formula for ␩. Ž3. The estimator ␪ˆ is used in the ␭ Behrens᎐Fisher test statistic for comparing the means of two populations with unknown variances ␴ 12 and ␴ 22 , which are assumed to be unequal. If Y1 and Y2 are the means of two independent samples of sizes n1 s 7 and n 2 s 9, respectively, randomly chosen from these populations, then ␪ is the variance of Y1 y Y2 . In this case, X 1 and X 2 represent the corresponding sample variances. The Behrens᎐Fisher t-statistic is then given by ts

Y1 y Y2

'␪ˆ

.

If the two population means are equal, t has approximately the t-distribution with ␩ degrees of freedom. For more details about the Behrens᎐Fisher test, see for example, Brownlee Ž1965, Section 9.8..x

325

EXERCISES

7.37. Suppose that a parabola of the form ␮ s ␤ 0 q ␤ 1 xq ␤ 2 x 2 is fitted to a set of paired data, Ž x 1 , y 1 ., Ž x 2 , y 2 ., . . . , Ž x n , yn .. Obtain estimates of ␤ 0 , ␤ 1 , and ␤ 2 by minimizing Ý nis1 w yi y Ž ␤ 0 q ␤ 1 x i q ␤ 2 x i2 .x2 with respect to ␤ 0 , ␤ 1 , and ␤ 2 . w Note: The estimates obtained in this manner are the least-squares estimates of ␤ 0 , ␤ 1 , and ␤ 2 .x 7.38. Suppose that we have k disjoint events A1 , A 2 , . . . , A k such that the probability of A i is pi Ž i s 1, 2, . . . , k . and Ý kis1 pi s 1. Furthermore, suppose that among n independent trials there are X 1 , X 2 , . . . , X k outcomes associated with A1 , A 2 , . . . , A k , respectively. The joint probability that X 1 s x 1 , X 2 s x 2 , . . . , X k s x k is given by the likelihood function L Ž x, p . s

n! x 1 ! x 2 ! ⭈⭈⭈ x k !

p1x 1 p 2x 2 ⭈⭈⭈ pkx k ,

where x i s 0, 1, 2, . . . , n for i s 1, 2, . . . , k such that Ý kis1 x i s n, x s Ž x 1 , x 2 , . . . , x k .⬘, p s Ž p1 , p 2 , . . . , pk .⬘. This defines a joint distribution for X 1 , X 2 , . . . , X k known as the multinomial distribution. Find the maximum likelihood estimates of p1 , p 2 , . . . , pk by maximizing LŽx, p. subject to Ý kis1 pi s 1. w Hint: Maximize the natural logarithm of p1x 1 p 2x 2 ⭈⭈⭈ pkx k subject to Ý kis1 pi s 1.x 7.39. Let ␾ Ž y . be a positive, even, and continuous function on Žy⬁, ⬁. such ⬁ ␾ Ž y . dy s 1. Consider that ␾ Ž y . is strictly decreasing on Ž0, ⬁., and Hy⬁ the following bivariate density function:

°1 q xr␾ Ž y . , ¢0

y␾ Ž y . F x- 0, 0FxF␾ Ž y. , otherwise.

f Ž x, y . s~1 y xr␾ Ž y . ,

(a) Show that f Ž x, y . is continuous for y⬁ - x, y - ⬁. (b) Let F Ž x, y . be the corresponding cumulative distribution function, F Ž x, y . s

Hy⬁Hy⬁ f Ž s, t . ds dt. x

y

Show that if 0 - ⌬ x- ␾ Ž0., then F Ž ⌬ x, 0 . y F Ž 0, 0 . G

H0␾

y1 Ž

⌬ x.

H0

⌬x

1y

s

␾Ž t.

ds dt

G 12 ⌬ x ␾y1 Ž ⌬ x . , where ␾y1 is the inverse function of ␾ Ž y . for 0 F y - ⬁.

326

MULTIDIMENSIONAL CALCULUS

(c) Use part Žb. to show that lim q

F Ž ⌬ x, 0 . y F Ž 0, 0 .

⌬ x™0

⌬x

s ⬁.

Hence, ⭸ F Ž x, y .r⭸ x does not exist at Ž0, 0.. (d) Deduce from part Žc. that the equality f Ž x, y . s

⭸ 2 F Ž x, y . ⭸x⭸y

does not hold in this example. w Note: This example was given by Wen Ž2001. to demonstrate that continuity of f Ž x, y . is not sufficient for the existence of ⭸ Fr⭸ x, and hence for the validity of the equality in part Žd..x

CHAPTER 8

Optimization in Statistics

Optimization is an essential feature in many problems in statistics. This is apparent in almost all fields of statistics. Here are few examples, some of which will be discussed in more detail in this chapter. 1. In the theory of estimation, an estimator of an unknown parameter is sought that satisfies a certain optimality criterion such as minimum variance, maximum likelihood, or minimum average risk Žas in the case of a Bayes estimator.. Some of these criteria were already discussed in Section 7.11. For example, in regression analysis, estimates of the parameters of a fitted model are obtained by minimizing a certain expression that measures the closeness of the fit of the model. One common example of such an expression is the sum of the squared residuals Žthese are deviations of the predicted response values, as specified by the model, from the corresponding observed response values.. This particular expression is used in the method of ordinary least squares. A more general class of parameter estimators is the class of M-estimators. See Huber Ž1973, 1981.. The name ‘‘ M-estimator’’ comes from ‘‘generalized maximum likelihood.’’ They are based on the idea of replacing the squared residuals by another symmetric function of the residuals that has a unique minimum at zero. For example, minimizing the sum of the absolute values of the residuals produces the so-called least absolute ®alues ŽLAV. estimators. 2. Estimates of the variance components associated with random or mixed models are obtained by using several methods. In some of these methods, the estimates are given as solutions to certain optimization problems as in maximum likelihood ŽML. estimation and minimum norm quadratic unbiased estimation ŽMINQUE.. In the former method, the likelihood function is maximized under the assumption of normally distributed data wsee Hartley and Rao Ž1967.x. A completely different approach is used in the latter method, which was proposed by Rao Ž1970, 1971.. This method does not require the normality assumption. 327

328

3.

4.

5.

6.

7.

OPTIMIZATION IN STATISTICS

For a review of methods of estimating variance components, see Khuri and Sahai Ž1985.. In statistical inference, tests are constructed so that they are optimal in a certain sense. For example, in the Neyman᎐Pearson lemma Žsee, for example, Roussas, 1973, Chapter 13., a test is obtained by minimizing the probability of Type II error while holding the probability of Type I error at a certain level. In the field of response surface methodology, design settings are chosen to minimize the prediction variance inside a region of interest, or to minimize the bias that occurs from fitting the ‘‘ wrong’’ model. Other optimality criteria can also be considered. For example, under the D-optimality criterion, the determinant of the variance᎐covariance matrix of the least-squares estimator of the vector of unknown parameters Žof a fitted model. is minimized with respect to the design settings. Another objective of response surface methodology is the determination of optimum operating conditions on the input variables that produce maximum, or minimum, response values inside a region of interest. For example, in a particular chemical reaction setting, it may be of interest to determine the reaction temperature and the reaction time that maximize the percentage yield of a product. Optimum seeking methods in response surface methodology will be discussed in detail in Section 8.3. Several response variables may be observed in an experiment for each setting of a group of input variables. Such an experiment is called a multiresponse experiment. In this case, optimization involves a number of response functions and is therefore referred to as simultaneous Žor multiresponse . optimization. For example, it may be of interest to maximize the yield of a certain chemical compound while reducing the production cost. Multiresponse optimization will be discussed in Section 8.7. In multivariate analysis, a large number of measurements may be available as a result of some experiment. For convenience in the analysis and interpretation of such data, it would be desirable to work with fewer of the measurements, without loss of much information. This problem of data reduction is dealt with by choosing certain linear functions of the measurements in an optimal manner. Such linear functions are called principal components. Optimization of a multivariable function was discussed in Chapter 7. However, there are situations in which the optimum cannot be obtained explicitly by simply following the methods described in Chapter 7. Instead, iterative procedures may be needed. In this chapter, we shall first discuss some commonly used iterative optimization methods. A number of these methods require the explicit evaluation of the partial derivatives of the function to be optimized Žobjective function.. These

329

THE GRADIENT METHODS

are referred to as the gradient methods. Three other optimization techniques that rely solely on the values of the objective function will also be discussed. They are called direct search methods.

8.1. THE GRADIENT METHODS Let f Žx. be a real-valued function of k variables x 1 , x 2 , . . . , x k , where x s Ž x 1 , x 2 , . . . , x k .⬘. The gradient methods are based on approximating f Žx. with a low-degree polynomial, usually of degree one or two, using Taylor’s expansion. The first- and second-order partial derivatives of f Žx. are therefore assumed to exist at every point x in the domain of f. Without loss of generality, we shall consider that f is to be minimized. 8.1.1. The Method of Steepest Descent This method is based on a first-order approximation of f Žx. with a polynomial of degree one using Taylor’s theorem Žsee Section 7.5.. Let x 0 be an initial point in the domain of f Žx.. Let x 0 q t h 0 be a neighboring point, where t h 0 represents a small change in the direction of a unit vector h 0 Žthat is, t ) 0.. The corresponding change in f Žx. is f Žx 0 q t h 0 . y f Žx 0 .. A firstorder approximation of this change is given by f Ž x 0 q t h 0 . y f Ž x 0 . f t hX0⵱f Ž x 0 . ,

Ž 8.1 .

as can be seen from applying formula Ž7.27.. If the objective is to minimize f Žx., then h 0 must be chosen so as to obtain the largest value for ythX0⵱f Žx 0 .. This is a constrained maximization problem, since h 0 has unit length. For this purpose we use the method of Lagrange multipliers. Let F be the function F s ythX0⵱f Ž x 0 . q ␭ Ž hX0 h 0 y 1 . . By differentiating F with respect to the elements of h 0 and equating the derivatives to zero we obtain h0 s

t 2␭

⵱f Ž x 0 . .

Ž 8.2 .

Using the constraint hX0 h 0 s 1, we find that ␭ must satisfy the equation

␭2 s

t2 4

5 ⵱f Ž x 0 . 5 22 ,

Ž 8.3 .

where 5 ⵱f Žx 0 .5 2 is the Euclidean norm of ⵱f Žx 0 .. In order for ythX0⵱f Žx 0 .

330

OPTIMIZATION IN STATISTICS

to have a maximum, ␭ must be negative. From formula Ž8.3. we then have t ␭ s y 5 ⵱f Ž x 0 . 5 2 . 2 By substituting this expression in formula Ž8.2. we get h0 sy

⵱f Ž x 0 . 5 ⵱f Ž x 0 . 5 2

Ž 8.4 .

.

Thus for a given t ) 0, we can achieve a maximum reduction in f Žx 0 . by moving from x 0 in the direction specified by h 0 in formula Ž8.4.. The value of t is now determined by performing a linear search in the direction of h 0 . This is accomplished by increasing the value of t Žstarting from zero. until no further reduction in the values of f is obtained. Let such a value of t be denoted by t 0 . The corresponding value of x is given by x 1 s x 0 y t0

⵱f Ž x 0 . 5 ⵱f Ž x 0 . 5 2

.

Since the direction of h 0 is in general not toward the location x* of the true minimum of f, the above process must be performed iteratively. Thus if at stage i we have an approximation x i for x*, then at stage i q 1 we have the approximation x iq1 s x i q t i h i ,

i s 0, 1, 2, . . . ,

where hisy

⵱f Ž x i . 5 ⵱f Ž x i . 5 2

,

i s 0, 1, 2, . . . ,

and t i is determined by a linear search in the direction of h i , that is, t i is the value of t that minimizes f Žx i q t h i .. Note that if it is desired to maximize f, then for each i ŽG 0. we need to move in the direction of yh i . In this case, the method is called the method of steepest ascent. Convergence of the method of steepest descent can be very slow, since frequent changes of direction may be necessary. Another reason for slow convergence is that the direction of h i at the ith iteration may be nearly perpendicular to the direction toward the minimum. Furthermore, the method becomes inefficient when the first-order approximation of f is no longer adequate. In this case, a second-order approximation should be attempted. This will be described in the next section.

331

THE GRADIENT METHODS

8.1.2. The Newton–Raphson Method Let x 0 be an initial point in the domain of f Žx.. By a Taylor’s expansion of f in a neighborhood of x 0 Žsee Theorem 7.5.1., it is possible to approximate f Žx. with the quadratic function ␾ Žx. given by

␾ Ž x . s f Ž x 0 . q Ž x y x 0 . ⬘⵱f Ž x 0 . q

1 2!

Ž x y x 0 . ⬘H f Ž x 0 . Ž x y x 0 . , Ž 8.5 .

where H f Žx 0 . is the Hessian matrix of f evaluated at x 0 . On the basis of formula Ž8.5. we can obtain a reasonable approximation to the minimum of f Žx. by using the minimum of ␾ Žx.. If ␾ Žx. attains a local minimum at x 1 , then we must necessarily have ⵱␾ Žx 1 . s 0 Žsee Section 7.7., that is, ⵱f Ž x 0 . q H f Ž x 0 . Ž x 1 y x 0 . s 0.

Ž 8.6 .

If H f Žx 0 . is nonsingular, then from equation Ž8.6. we obtain x 1 s x 0 y Hy1 f Ž x 0 . ⵱f Ž x 0 . . If we now approximate f Žx. with another quadratic function, by again applying Taylor’s expansion in a neighborhood of x 1 , and then repeat the same process as before with x 1 used instead of x 0 , we obtain the point x 2 s x 1 y Hy1 f Ž x 1 . ⵱f Ž x 1 . . Further repetitions of this process lead to a sequence of points, x 0 , x 1 , x 2 , . . . , x i , . . . , such that x iq1 s x i y Hy1 f Ž x i . ⵱f Ž x i . ,

i s 0, 1, 2, . . . .

Ž 8.7 .

The Newton᎐Raphson method requires finding the inverse of the Hessian matrix H f at each iteration. This can be computationally involved, especially if the number of variables, k, is large. Furthermore, the method may fail to converge if H f Žx i . is not positive definite. This can occur, for example, when x i is far from the location x* of the true minimum. If, however, the initial point x 0 is close to x*, then convergence occurs at a rapid rate. 8.1.3. The Davidon–Fletcher–Powell Method This method is basically similar to the one in Section 8.1.1 except that at the ith iteration we have x iq1 s x i y ␪ i G i⵱f Ž x i . ,

i s 0, 1, 2, . . . ,

332

OPTIMIZATION IN STATISTICS

where G i is a positive definite matrix that serves as the ith approximation to the inverse of the Hessian matrix H f Žx i ., and ␪ i is a scalar determined by a linear search from x i in the direction of yG i⵱f Žx i ., similar to the one for the steepest descent method. The initial choice G 0 of the matrix G can be any positive definite matrix, but is usually taken to be the identity matrix. At the Ž i q 1.st iteration, G i is updated by using the formula G iq1 s G i q L i q M i ,

i s 0, 1, 2, . . . ,

where Lisy Misy

G i ⵱f Ž x iq1 . y ⵱f Ž x i .

⵱f Ž x iq1 . y ⵱f Ž x i . ⬘G i

⵱f Ž x iq1 . y ⵱f Ž x i . ⬘G i ⵱f Ž x iq1 . y ⵱f Ž x i .

␪ i G i⵱f Ž x i . G i⵱f Ž x i . ⬘ G i⵱f Ž x i . ⬘ ⵱f Ž x iq1 . y ⵱f Ž x i .

,

.

The justification for this method is given in Fletcher and Powell Ž1963.. See also Bunday Ž1984, Section 4.3.. Note that if G i is initially chosen as the identity, then the first increment is in the steepest descent direction y⵱f Žx 0 .. This is a powerful optimization method and is considered to be very efficient for most functions.

8.2. THE DIRECT SEARCH METHODS The direct search methods do not require the evaluation of any partial derivatives of the objective function. For this reason they are suited for situations in which it is analytically difficult to provide expressions for the partial derivatives, such as the minimization of the maximum absolute deviation. Three such methods will be discussed here, namely, the Nelder᎐Mead simplex method, Price’s controlled random search procedure, and generalized simulated annealing. 8.2.1. The Nelder–Mead Simplex Method Let f Žx., where x s Ž x 1 , x 2 , . . . , x k .⬘, be the function to be minimized. The simplex method is based on a comparison of the values of f at the k q 1 vertices of a general simplex followed by a move away from the vertex with the highest function value. By definition, a general simplex is a geometric figure formed by a set of k q 1 points called vertices in a k-dimensional space. Originally, the simplex method was proposed by Spendley, Hext, and Himsworth Ž1962., who considered a regular simplex, that is, a simplex with mutually equidistant points such as an equilateral triangle in a two-dimensional space Ž k s 2.. Nelder and Mead Ž1965. modified this method by

333

THE DIRECT SEARCH METHODS

allowing the simplex to be nonregular. This modified version of the simplex method will be described here. The simplex method follows a sequential search procedure. As was mentioned earlier, it begins by evaluating f at the k q 1 points that form a general simplex. Let these points be denoted by x 1 , x 2 , . . . , x kq1 . Let f h and f l denote, respectively, the largest and the smallest of the values f Žx 1 ., f Žx 2 ., . . . , f Žx kq1 .. Let us also denote the points where f h and f l are attained by x h and x l , respectively. Obviously, if we are interested in minimizing f, then a move away from x h will be in order. Let us therefore define x c as the centroid of all the points with the exclusion of x h . Thus xcs

1 k

Ý xi.

i/h

In order to move away from x h , we reflect x h with respect to x c to obtain the point x Uh . More specifically, the latter point is defined by the relation x Uh y x c s r Ž x c y x h . , or equivalently, x Uh s Ž 1 q r . x c y r x h , where r is a positive constant called the reflection coefficient and is given by rs

5x Uh y x c 5 2 5x c y x h 5 2

.

The points x h , x c , and x Uh are depicted in Figure 8.1. Let us consider the

Figure 8.1. A two-dimensional simplex with the reflection ŽxUh ., expansion ŽxUh e ., and contraction ŽxUh c . points.

334

OPTIMIZATION IN STATISTICS

following cases: a. If f l - f Žx Uh . - f h , replace x h by x Uh and start the process again with the new simplex Žthat is, evaluate f at the vertices of the simplex which has the same points as the original simplex, but with x Uh substituted for x h .. b. If f Žx Uh . - f l , then the move from x c to x Uh is in the right direction and should therefore be expanded. In this case, x Uh is expanded to x Uh e defined by the relation x Uh e y x c s ␥ Ž x Uh y x c . , that is, x Uh e s ␥ x Uh q Ž 1 y ␥ . x c , where ␥ Ž) 1. is an expansion coefficient given by

␥s

5 x Uh e y x c 5 2 5 x Uh y x c 5 2

Žsee Figure 8.1.. This operation is called expansion. If f Žx Uh e . - f l , replace x h by x Uh e and restart the process. However, if f Žx Uh e . ) f l , then expansion is counterproductive. In this case, x Uh e is dropped, x h is replaced by x Uh , and the process is restarted. c. If upon reflecting x h to x Uh we discover that f Žx Uh . ) f Žx i . for all i / h, then replacing x h by x Uh would leave f Žx Uh . as the maximum in the new simplex. In this case, a new x h is defined to be either the old x h or x Uh , whichever has the lower value. A point x Uh c is then found such that x Uh c y x c s ␤ Ž x h y x c . , that is, x Uh c s ␤ x h q Ž 1 y ␤ . x c , where ␤ Ž0 - ␤ - 1. is a contraction coefficient given by

␤s

5 x Uh c y x c 5 2 5 xhyxc52

.

Next, x Uh c is substituted for x h and the process is restarted unless f Žx Uh c . ) minw f h , f Žx Uh .x, that is, the contracted point is worse than the better of f h and f Žx Uh .. When such a contraction fails, the size of the simplex is reduced by halving the distance of each point of the simplex from x l , where, if we recall, x l is the point generating the lowest function value. Thus x i is replaced by x l q 12 Žx i y x l ., that is, by 12 Žx i q x l .. The process is then restarted with the new reduced simplex.

THE DIRECT SEARCH METHODS

335

Figure 8.2. Flow diagram for the Nelder᎐Mead simplex method. Source: Nelder and Mead Ž1965.. Reproduced with permission of Oxford University Press.

Thus at each stage in the minimization process, x h , the point at which f has the highest value, is replaced by a new point according to one of three operations, namely, reflection, contraction, and expansion. As an aid to illustrating this step-by-step procedure, a flow diagram is shown in Figure 8.2. This flow diagram is similar to one given by Nelder and Mead Ž1965, page 309.. Figure 8.2 lists the explanations of steps 1 through 6. The criterion used to stop the search procedure is based on the variation in the function values over the simplex. At each step, the

336

OPTIMIZATION IN STATISTICS

standard error of these values in the form

ss

Ý kq1 is1 Ž f i y f .

2

1r2

k

is calculated and compared with some preselected value d, where f 1 , f 2 , . . . , f kq1 denote the function values at the vertices of the simplex Ž . at hand and f s Ý kq1 is1 f ir k q 1 . The search is halted when s - d. The reasoning behind this criterion is that when s - d, all function values are very close together. This hopefully indicates that the points of the simplex are near the minimum. Bunday Ž1984. provided the listing of a computer program which can be used to implement the steps described in the flow diagram. Olsson and Nelson Ž1975. demonstrated the usefulness of this method by using it to solve six minimization problems in statistics. The robustness of the method itself and its advantages relative to other minimization techniques were reported in Nelson Ž1973.. 8.2.2. Price’s Controlled Random Search Procedure The controlled random search procedure was introduced by Price Ž1977.. It is capable of finding the absolute Žor global. minimum of a function within a constrained region R. It is therefore well suited for a multimodal function, that is, a function that has several local minima within the region R. The essential features of Price’s algorithm are outlined in the flow diagram of Figure 8.3. A predetermined number, N, of trial points are randomly chosen inside the region R. The value of N must be greater than k, the number of variables. The corresponding function values are obtained and stored in an array A along with the coordinates of the N chosen points. At each iteration, k q 1 distinct points, x 1 , x 2 , . . . , x kq1 , are chosen at random from the N points in storage. These k q 1 points form a simplex in a k-dimensional space. The point x kq1 is arbitrarily taken as the pole Ždesignated vertex. of the simplex, and the next trial point x t is obtained as the image Žreflection. point of the pole with respect to the centroid x c of the remaining k points. Thus x t s 2x c y x kq1 . The point x t must satisfy the constraints of the region R. The value of the function f at x t is then compared with f max , the largest function value in storage. Let x max denote the point at which f max is achieved. If f Žx t . - f max , then x max is replaced in the array A by x t . If x t fails to satisfy the constraints of the region R, or if f Žx t . ) f max , then x t is discarded and a new point is

THE DIRECT SEARCH METHODS

337

Figure 8.3. A flow diagram for Price’s procedure. Source: Price Ž1977.. Reproduced with permission of Oxford University Press.

338

OPTIMIZATION IN STATISTICS

chosen by following the same procedure as the one used to obtain x t . As the algorithm proceeds, the N points in storage tend to cluster around points at which the function values are lower than the current value of f max . Price did not specify a particular stopping rule. He left it to the user to do so. A possible stopping criterion is to terminate the search when the N points in storage cluster in a small region of the k-dimensional space, that is, when f max and f min are close together, where f min is the smallest function value in storage. Another possibility is to stop after a specified number of function evaluations have been made. In any case, the rate of convergence of the procedure depends on the value of N, the complexity of the function f, the nature of the constraints, and the way in which the set of trial points is chosen. Price’s procedure is simple and does not necessarily require a large value of N. It is sufficient that N should increase linearly with k. Price chose, for example, the value N s 50 for k s 2. The value N s 10 k has proved useful for many functions. Furthermore, the region constraints can be quite complex. A FORTRAN program for the implementation of Price’s algorithm was written by Conlon Ž1991.. 8.2.3. The Generalized Simulated Annealing Method This method derives its name from the annealing of metals, in which many final crystalline configurations Žcorresponding to different energy states . are possible, depending on the rate of cooling Žsee Kirkpartrick, Gelatt, and Vechhi, 1983.. The method can be applied to find the absolute Žor global. optimum of a multimodal function f within a constrained region R in a k-dimensional space. Bohachevsky, Johnson, and Stein Ž1986. presented a generalization of the method of simulated annealing for function optimization. The following is a description of their algorithm for function minimization Ža similar one can be used for function maximization.: Let f m be some tentative estimate of the minimum of f over the region R. The method proceeds according to the following steps Žreproduced with permission of the American Statistical Association.: 1. Select an initial point x 0 in R. This point can be chosen at random or specified depending on available information. 2. Calculate f 0 s f Žx 0 .. If < f 0 y f m < - ⑀ , where ⑀ is a specified small constant, then stop. 3. Choose a random direction of search by generating k independent standard normal variates z1 , z 2 , . . . , z k ; then compute the elements of the random vector u s Ž u1 , u 2 , . . . , u k .⬘, where zi ui s , i s 1, 2, . . . , k, 1r2 Ž z12 q z 22 q ⭈⭈⭈ qz k2 . and k is the number of variables in the function.

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

339

4. Set x 1 s x 0 q ⌬ r u, where ⌬ r is the size of a step to be taken in the direction of u. The magnitude of ⌬ r depends on the properties of the objective function and on the desired accuracy. 5. If x 1 does not belong to R, return to step 3. Otherwise, compute f 1 s f Žx 1 . and ⌬ f s f 1 y f 0 . 6. if f 1 F f 0 , set x 0 s x 1 and f 0 s f 1. If < f 0 y f m < - ⑀ , stop. Otherwise, go to step 3. 7. If f 1 ) f 0 , set a probability value given by ps expŽy␤ f 0g ⌬ f ., where ␤ is a positive number such that 0.50 - expŽy␤ ⌬ f . - 0.90, and g is an arbitrary negative number. Then, generate a random number ® from the uniform distribution U Ž0, 1.. If ®G p, go to step 3. Otherwise, if ®- p, set x 0 s x 1 , f 0 s f 1 , and go to step 3. From steps 6 and 7 we note that beneficial steps Žthat is, f 1 F f 0 . are accepted unconditionally, but detrimental steps Ž f 1 ) f 0 . are accepted according to a probability value p described in step 7. If ®- p, then the step leading to x 1 is accepted; otherwise, it is rejected and a step in a new random direction is attempted. Thus the probability of accepting an increment of f depends on the size of the increment: the larger the increment, the smaller the probability of its acceptance. Several possible values of the tentative estimate f m can be attempted. For a given f m we proceed with the search until f y f m becomes negative. Then, we decrease f m , continue the search, and repeat the process when necessary. Bohachevsky, Johnson, and Stein gave an example in optimal design theory to illustrate the application of their algorithm. Price’s Ž1977. controlled random search algorithm produces results comparable to those of simulated annealing, but with fewer tuning parameters. It is also better suited for problems with constrained regions.

8.3. OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY Response surface methodology ŽRSM. is an area in the design and analysis of experiments. It consists of a collection of techniques that encompasses: 1. Conducting a series of experiments based on properly chosen settings of a set of input variables, denoted by x 1 , x 2 , . . . , x k , that influence a response of interest y. The choice of these settings is governed by certain criteria whose purpose is to produce adequate and reliable information about the response. The collection of all such settings constitutes a matrix D of order n = k, where n is the number of experimental runs. The matrix D is referred to as a response surface design.

340

OPTIMIZATION IN STATISTICS

2. Determining a mathematical model that best fits the data collected under the design chosen in Ž1.. Regression techniques can be used to evaluate the adequacy of fit of the model and to conduct appropriate tests concerning the model’s parameters. 3. Determining optimal operating conditions on the input variables that produce maximum Žor minimum. response value within a region of interest R. This last aspect of RSM can help the experimenter in determining the best combinations of the input variables that lead to desirable response values. For example, in drug manufacturing, two drugs are tested with regard to reducing blood pressure in humans. A series of clinical trials involving a certain number of high blood pressure patients is set up, and each patient is given some predetermined combination of the two drugs. After a period of time the patient’s blood pressure is checked. This information can be used to find the specific combination of the drugs that results in the greatest reduction in the patient’s blood pressure within some specified time interval. In this section we shall describe two well-known optimum-seeking procedures in RSM. These include the method of steepest ascent Žor descent . and ridge analysis.

8.3.1. The Method of Steepest Ascent This is an adaptation of the method described in Section 8.1.1 to a response surface environment; here the objective is to increase the value of a certain response function. The method of steepest ascent requires performing a sequence of sets of trials. Each set is obtained as a result of proceeding sequentially along a path of maximum increase in the values of a given response y, which can be observed in an experiment. This method was first introduced by Box and Wilson Ž1951. for the general area of RSM. The procedure of steepest ascent depends on approximating a response surface with a hyperplane in some restricted region. The hyperplane is represented by a first-order model which can be fitted to a data set obtained as a result of running experimental trials using a first-order design such as a complete 2 k factorial design, where k is the number of input variables in the model. A fraction of this design can also be used if k is large wsee, for example, Section 3.3.2 in Khuri and Cornell Ž1996.x. The fitted first-order model is then used to determine a path along which one may initially observe increasing response values. However, due to curvature in the response surface, the initial increase in the response will likely be followed by a leveling off, and then a decrease. At this stage, a new series of experiments is performed Žusing again a first-order design. and the resulting data are used to fit another first-order model. A new path is determined along which

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

341

increasing response values may be observed. This process continues until it becomes evident that little or no additional increase in the response can be gained. Let us now consider more specific details of this sequential procedure. Let y Žx. be a response function that depends on k input variables, x 1 , x 2 , . . . , x k , which form the elements of a vector x. Suppose that in some restricted region y Žx. is adequately represented by a first-order model of the form y Ž x. s ␤0 q

k

Ý ␤i x i q ⑀ ,

Ž 8.8 .

is1

where ␤ 0 , ␤ 1 , . . . , ␤ k are unknown parameters and ⑀ is a random error. This model is fitted using data collected under a first-order design Žfor example, a 2 k factorial design or a fraction thereof.. The data are utilized to calculate the least-squares estimates ␤ˆ0 , ␤ˆ1 , . . . , ␤ˆk of the model’s parameters. These ˆ s ŽX⬘X.y1 X⬘y, where X s w1 n : Dx with 1 n being a vector of are elements of ␤ ones of order n = 1, D is the design matrix of order n = k, and y is the corresponding vector of response values. It is assumed that the random errors associated with the n response values are independently distributed with means equal to zero and a common variance ␴ 2 . The predicted response ˆ y Žx. is then given by k

ˆy Ž x . s ␤ˆ0 q Ý ␤ˆi x i .

Ž 8.9 .

is1

The input variables are coded so that the design center coincides with the origin of the coordinates system. The next step is to move a distance of r units away from the design center Žor the origin. such that a maximum increase in ˆ y can be obtained. To determine the direction to be followed to achieve such an increase, we need to maximize ˆ y Žx. subject to the constraint Ý kis1 x i2 s r 2 using the method of Lagrange multipliers. Consider therefore the function Q Ž x . s ␤ˆ0 q

ž

k

k

Ý ␤ˆi x i y ␭ Ý x i2 y r 2

is1

is1

/

,

Ž 8.10 .

where ␭ is a Lagrange multiplier. Setting the partial derivatives of Q equal to zero produces the equations xi s

1 2␭

␤ˆi ,

i s 1, 2, . . . , k.

For a maximum, ␭ must be positive. Using the equality constraint, we conclude that

␭s

1 2r

ž / k

Ý ␤ˆi2

is1

1r2

.

342

OPTIMIZATION IN STATISTICS

A local maximum is then achieved at the point whose coordinates are given by xi s

r ␤ˆi

Ž Ýkis1 ␤ˆi2 .

1r2

,

i s 1, 2, . . . , k,

which can be written as x i s re i ,

i s 1, 2, . . . , k,

Ž 8.11 .

where e i s ␤ˆi ŽÝ kis1 ␤ˆi2 .y1r2 , i s 1, 2, . . . , k. Thus e s Ž e1 , e2 , . . . , e k .⬘ is a unit vector in the direction of Ž ␤ˆ1 , ␤ˆ2 , . . . , ␤ˆk .⬘. Equations Ž8.11. indicate that at a distance of r units away from the origin, a maximum increase in ˆ y occurs along a path in the direction of e. Since this is the only local maximum on the hypersphere of radius r, it must be the absolute maximum. If the actual response value Žthat is, the value of y . at the point x s r e exceeds its value at the origin, then a move along the path determined by e is in order. A series of experiments is then conducted to obtain response values at several points along the path until no additional increase in the response is evident. At this stage, a new first-order model is fitted using data collected under a first-order design centered at a point in the vicinity of the point at which that first drop in the response was observed along the path. This model leads to a new direction similar to the one given by formula Ž8.11.. As before, a series of experiments are conducted along the new path until no further increase in the value of y can be observed. The process of moving along different paths continues until it becomes evident that little or no additional increase in y can be gained. This usually occurs when the first-order model becomes inadequate as the method progresses, due to curvature in the response surface. It is therefore necessary to test each fitted first-order model for lack of fit at every stage of the process. This can be accomplished by taking repeated observations at the center of each first-order design and at possibly some other design points in order to obtain an independent estimate of the error variance that is needed for the lack of fit test wsee, for example, Sections 2.6 and 3.4 in Khuri and Cornell Ž1996.x. If the lack of fit test is significant, indicating an inadequate model, then the process is stopped and a more elaborate experiment must be conducted to fit a higher-order model, as will be seen in the next section. Examples that illustrate the application of the method of steepest ascent can be found in Box and Wilson Ž1951., Bayne and Rubin Ž1986, Section 5.2., Khuri and Cornell Ž1996, Chapter 5., and Myers and Khuri Ž1979.. In the last reference, the authors present a stopping rule along a path that takes into account random error variation in the observed response. We recall that a search along a path is discontinued as soon as a drop in the response is first

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

343

observed. Since response values are subject to random error, the decision to stop can be premature due to a false drop in the observed response. The stopping rule by Myers and Khuri Ž1979. protects against taking too many observations along a path when in fact the true mean response Žthat is, the mean of y . is decreasing. It also protects against stopping prematurely when the true mean response is increasing. It should be noted that the procedure of steepest ascent is not invariant with respect to the scales of the input variables x 1 , x 2 , . . . , x k . This is evident from the fact that a path taken by the procedure is determined by the least-squares estimates ␤ˆ1 , ␤ˆ2 , . . . , ␤ˆk wsee equations Ž8.11.x, which depend on the scales of the x i ’s. There are situations in which it is of interest to determine conditions that lead to a decrease in the response, instead of an increase. For example, in a chemical investigation it may be desired to decrease the level of impurity or the unit cost. In this case, a path of steepest descent will be needed. This can be accomplished by changing the sign of the response y, followed by an application of the method of steepest ascent. Thus any steepest descent problem can be handled by the method of steepest ascent.

8.3.2. The Method of Ridge Analysis The method of steepest ascent is most often used as a maximum-region-seeking procedure. By this we mean that it is used as a preliminary tool to get quickly to the region where the maximum of the mean response is located. Since the first-order approximation of the mean response will eventually break down, a better estimate of the maximum can be obtained by fitting a second-order model in the region of the maximum. The method of ridge analysis, which was introduced by Hoerl Ž1959. and formalized by Draper Ž1963., is used for this purpose. Let us suppose that inside a region of interest R, the true mean response is adequately represented by the second-order model

y Ž x. s ␤0 q

k

Ý

is1

ky1

␤i x i q

k

Ý Ý

k

␤i j x i x j q

is1 js2

Ý ␤ii x i2 q ⑀ ,

Ž 8.12 .

is1

i-j

where the ␤ ’s are unknown parameters and ⑀ is a random error with mean zero and variance ␴ 2 . Model Ž8.12. can be written as y Ž x . s ␤ 0 q x⬘␤ q x⬘Bx q ⑀ ,

Ž 8.13 .

344

OPTIMIZATION IN STATISTICS

where ␤ s Ž ␤ 1 , ␤ 2 , . . . , ␤ k .⬘ and B is a symmetric k = k matrix of the form

␤ 11

1 2

␤ 12

␤ 22

1 2

␤ 13

⭈⭈⭈

1 2

␤1 k

1 2

␤ 23

⭈⭈⭈ .. .

1 2

␤2 k . . .

.. .

Bs

.. . symmetric

1 2

.

␤ ky1 , k ␤k k

Least-squares estimates of the parameters in model Ž8.13. can be obtained by using data collected according to a second-order design. A description of potential second-order designs can be found in Khuri and Cornell Ž1996, Chapter 4.. ˆ and B ˆ denote the least-squares estimates of ␤ 0 , ␤, and B, Let ␤ˆ0 , ␤, respectively. The predicted response ˆ y Žx. inside the region R is then given by

ˆ q x⬘Bx. ˆ ˆy Ž x . s ␤ˆ0 q x⬘␤

Ž 8.14 .

The input variables are coded so that the design center coincides with the origin of the coordinates system. The method of ridge analysis is used to find the optimum Žmaximum or minimum. of ˆ y Žx. on concentric hyperspheres of varying radii inside the region R. It is particularly useful in situations in which the unconstrained optimum of ˆ y Žx. falls outside the region R, or if a saddle point occurs inside R. Let us now proceed to optimize ˆ y Žx. subject to the constraint k

Ý x i2 s r 2 ,

Ž 8.15 .

is1

where r is the radius of a hypersphere centered at the origin and is contained inside the region R. Using the method of Lagrange multipliers, let us consider the function F sˆ y Ž x. y ␭

žÝ k

is1

/

x i2 y r 2 ,

Ž 8.16 .

where ␭ is a Lagrange multiplier. Differentiating F with respect to x i

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

345

Ž i s 1, 2, . . . , k . and equating the partial derivatives to zero, we obtain

⭸F ⭸ x1 ⭸F ⭸ x2

s 2 Ž ␤ˆ11 y ␭ . x 1 q ␤ˆ12 x 2 q ⭈⭈⭈ q␤ˆ1 k x k q ␤ˆ1 s 0, s ␤ˆ12 x 1 q 2 Ž ␤ˆ22 y ␭ . x 2 q ⭈⭈⭈ q␤ˆ2 k x k q ␤ˆ2 s 0, . . .

⭸F ⭸ xk

s ␤ˆ1 k x 1 q ␤ˆ2 k x 2 q ⭈⭈⭈ q2 Ž ␤ˆk k y ␭ . x k q ␤ˆk s 0.

These equations can be expressed as

Ž Bˆ y ␭I k . x s y

1 2

ˆ ␤.

Ž 8.17 .

Equations Ž8.15. and Ž8.17. need to be solved for x 1 , x 2 , . . . , x k and ␭. This traditional approach, however, requires calculations that are somewhat involved. Draper Ž1963. proposed the following simpler, yet equivalent procedure: i. Regard r as a variable, but fix ␭ instead. ii. Insert the selected value of ␭ in equation Ž8.17. and solve for x. The solution is used in steps iii and iv. iii. Compute r s Žx X x.1r2 . iv. Evaluate ˆ y Žx.. Several values of ␭ can give rise to several stationary points which lie on the same hypersphere of radius r. This can be seen from the fact that if ␭ is ˆ then equation Ž8.17. has a chosen to be different from any eigenvalue of B, unique solution given by

ˆ y ␭I k . x s y 12 Ž B

y1

ˆ ␤.

Ž 8.18 .

ˆ s4r2. ␤

Ž 8.19 .

By substituting x in equation Ž8.15. we obtain

ˆ ŽB ˆ y ␭I k . ␤⬘

y2

Hence, each value of r gives rise to at most 2 k corresponding values of ␭. The choice of ␭ has an effect on the nature of the stationary point. Some values of ␭ produce points at each of which ˆ y has a maximum. Other values of ␭ cause ˆ y to have minimum values. More specifically, suppose that ␭1 and

346

OPTIMIZATION IN STATISTICS

␭2 are two values substituted for ␭ in equation Ž8.18.. Let x 1 , x 2 and r 1 , r 2 be the corresponding values of x and r, respectively. The following results, which were established in Draper Ž1963., can be helpful in selecting the value of ␭ that produces a particular type of stationary point: RESULT 1. If r 1 s r 2 and ␭1 ) ␭2 , then ˆ y1 )ˆ y 2 , where ˆ y 1 and ˆ y 2 are the values of ˆ y Žx. at x 1 and x 2 , respectively. This result means that for two stationary points that have the same distance from the origin, ˆ y will be larger at the stationary point with the larger value of ␭. RESULT 2. Let M be the matrix of second-order partial derivatives of F in formula Ž8.16., that is,

ˆ y ␭I k . . M s 2 ŽB

Ž 8.20 .

If r 1 s r 2 , and if M is positive definite for x 1 and is indefinite Žthat is, neither positive definite nor negative definite . for x 2 , then ˆ y1 -ˆ y2 .

ˆ then the RESULT 3. If ␭1 is larger than the largest eigenvalue of B, Ž . corresponding solution x 1 in formula 8.18 is a point of absolute maximum for ˆ y on a hypersphere of radius r 1 s Žx X1 x 1 .1r2 . If, on the other hand, ␭1 is ˆ then x 1 is a point of absolute smaller than the smallest eigenvalue of B, minimum for ˆ y on the same hypersphere. On the basis of Result 3 we can select several values of ␭ that exceed the ˆ The resulting values of the k elements of x and ˆy can largest eigenvalue of B. be plotted against the corresponding values of r. This produces k q 1 plots called ridge plots Žsee Myers, 1976, Section 5.3.. They are useful in that an experimenter can determine, for a particular r, the maximum of ˆ y within a region R and the operating conditions Žthat is, the elements of x. that give rise to the maximum. Similar plots can be obtained for the minimum of ˆ y ˆ must be Žhere, values of ␭ that are smaller than the smallest eigenvalue of B chosen.. Obviously, the portions of the ridge plots that fall outside R should not be considered. EXAMPLE 8.3.1. An experiment was conducted to investigate the effects of three fertilizer ingredients on the yield of snap beans under field conditions. The fertilizer ingredients and actual amounts applied were nitrogen ŽN., from 0.94 to 6.29 lbrplot; phosphoric acid ŽP2 O5 ., from 0.59 to 2.97 lbrplot; and potash ŽK 2 O., from 0.60 to 4.22 lbrplot. The response of interest is the average yield in pounds per plot of snap beans.

347

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

Five levels of each fertilizer were used. The levels are coded using the following linear transformations: X 1 y 3.62

x1 s

1.59

,

x2 s

X 2 y 1.78 0.71

,

x3 s

X 3 y 2.42 1.07

.

Here, X 1 , X 2 , and X 3 denote the actual levels of nitrogen, phosphoric acid, and potash, respectively, used in the experiment, and x 1 , x 2 , x 3 the corresponding coded values. In this particular coding scheme, 3.62, 1.78, and 2.42 are the averages of the experimental levels of X 1 , X 2 , and X 3 , respectively, that is, they represent the centers of the values of nitrogen, phosphoric acid, and potash, respectively. The denominators of x 1 , x 2 , and x 3 were chosen so that the second and fourth levels of each X i correspond to the values y1 and 1, respectively, for x i Ž i s 1, 2, 3.. One advantage of such a coding scheme is to make the levels of the three fertilizers scale free Žthis is necessary in general, since the input variables can have different units of measurement.. The measured and coded levels for the three fertilizers are shown below: Levels of x i Ž i s 1, 2, 3. Fertilizer N P2 O5 K 2O

y 1.682

y1.000

0.000

1.000

1.682

0.94 0.59 0.60

2.03 1.07 1.35

3.62 1.78 2.42

5.21 2.49 3.49

6.29 2.97 4.22

Combinations of the levels of the three fertilizers were applied according to the experimental design shown in Table 8.1, in which the design settings are given in terms of the coded levels. Six center-point replications were run in order to obtain an estimate of the experimental error variance. This particular design is called a central composite design wfor a description of this design and its properties, see Khuri and Cornell Ž1996, Section 4.5.3.x, which has the rotatability property. By this we mean that the prediction variance, that is, Varw ˆ y Žx.x, is constant at all points that are equidistant from the design w center see Khuri and Cornell Ž1996, Section 2.8.3. for more detailed information concerning rotatability x. The corresponding response Žyield. values are given in Table 8.1. A second-order model of the form given by formula Ž8.12. was fitted to the data set in Table 8.1. Thus in terms of the coded variables we have the model y Ž x. s ␤0 q

3

3

is1

is1

Ý ␤i x i q ␤12 x 1 x 2 q ␤13 x 1 x 3 q ␤ 23 x 2 x 3 q Ý ␤ii x i2 q ⑀ . Ž 8.21 .

348

OPTIMIZATION IN STATISTICS

Table 8.1. The Coded and Actual Settings of the Three Fertilizers and the Corresponding Response Values x1

x2

x3

N

P2 O5

K 2O

Yield y

y1 1 y1 1 y1 1 y1 1 y1.682 1.682 0 0 0 0 0 0 0 0 0 0

y1 y1 1 1 y1 y1 1 1 0 0 y1.682 1.682 0 0 0 0 0 0 0 0

y1 y1 y1 y1 1 1 1 1 0 0 0 0 y1.682 1.682 0 0 0 0 0 0

2.03 5.21 2.03 5.21 2.03 5.21 2.03 5.21 0.94 6.29 3.62 3.62 3.62 3.62 3.62 3.62 3.62 3.62 3.62 3.62

1.07 1.07 2.49 2.49 1.07 1.07 2.49 2.49 1.78 1.78 0.59 2.97 1.78 1.78 1.78 1.78 1.78 1.78 1.78 1.78

1.35 1.35 1.35 1.35 3.49 3.49 3.49 3.49 2.42 2.42 2.42 2.42 0.60 4.22 2.42 2.42 2.42 2.42 2.42 2.42

11.28 8.44 13.19 7.71 8.94 10.90 11.85 11.03 8.26 7.87 12.08 11.06 7.98 10.43 10.14 10.22 10.53 9.50 11.53 11.02

Source: A. I. Khuri and J. A. Cornell Ž1996.. Reproduced with permission of Marcel Dekker, Inc.

The resulting prediction equation is given by

ˆy Ž x . s 10.462 y 0.574 x 1 q 0.183 x 2 q 0.456 x 3 y 0.678 x 1 x 2 q 1.183 x 1 x 3 q 0.233 x 2 x 3 y 0.676 x 12 q 0.563 x 22 y 0.273 x 32 .

Ž 8.22 .

Here, ˆ y Žx. is the predicted yield at the point x s Ž x 1 , x 2 , x 3 .⬘. Equation Ž8.22. ˆs can be expressed in matrix form as in equation Ž8.14., where ␤ ˆ is the matrix Žy0.574, 0.183, 0.456.⬘ and B y0.676 ˆ s y0.339 B 0.592

y0.339 0.563 0.117

0.592 0.117 . y0.273

Ž 8.23 .

349

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

The coordinates of the stationary point x 0 of ˆ y Žx. satisfy the equation

⭸ˆ y ⭸ x1 ⭸ˆ y ⭸ x2 ⭸ˆ y ⭸ x3

s y0.574 q 2 Ž y0.676 x 1 y 0.339 x 2 q 0.592 x 3 . s 0, s 0.183 q 2 Ž y0.339 x 1 q 0.563 x 2 q 0.117x 3 . s 0, s 0.456 q 2 Ž 0.592 x 1 q 0.117x 2 y 0.273 x 3 . s 0,

which can be expressed as

Ž 8.24 .

ˆ q 2Bx ˆ 0 s 0. ␤ Hence,

ˆ y1 ␤ ˆ s Ž y0.394,y 0.364,y 0.175 . ⬘. x 0 s y 12 B ˆ are ␶ 1 s 0.6508, ␶ 2 s 0.1298, ␶ 3 s y1.1678. The matrix The eigenvalues of B ˆ B is therefore neither positive definite nor negative definite, that is, x 0 is a saddle point Žsee Corollary 7.7.1.. This point falls inside the experimental region R, which, in the space of the coded variables x 1 , x 2 , x 3 , is a sphere centered at the origin of radius '3 . Let us now apply the method of ridge analysis to maximize ˆ y inside the region R. For this purpose we choose values of ␭ wthe Lagrange multiplier in ˆ For each equation Ž8.16.x larger than ␶ 1 s 0.6508, the largest eigenvalue of B. such value of ␭, equation Ž8.17. has a solution for x that represents a point of absolute maximum of ˆ y Žx. on a sphere of radius r s Žx⬘x.1r2 inside R. The results are displayed in Table 8.2. We note that at the point Žy0.558, 1.640, 0.087., which is located near the periphery of the region R, the maximum value of ˆ y is 13.021. By expressing the coordinates of this point in terms of the actual values of the three fertilizers we obtain X 1 s 2.733 lbrplot, X 2 s 2.944 lbrplot, and X 3 s 2.513 lbrplot. We conclude that a combination of nitrogen, phosphoric acid, and potash fertilizers at the rates Table 8.2. Ridge Analysis Values ␭ x1 x2 x3 r ˆy

1.906 1.166 0.979 0.889 0.840 0.808 0.784 0.770 0.754 0.745 0.740 y0.106 y0.170 y0.221 y0.269 y0.316 y0.362 y0.408 y0.453 y0.499 y0.544 y0.558 0.102 0.269 0.438 0.605 0.771 0.935 1.099 1.263 1.426 1.589 1.640 0.081 0.110 0.118 0.120 0.117 0.113 0.108 0.102 0.096 0.089 0.087 0.168 0.337 0.505 0.673 0.841 1.009 1.177 1.346 1.514 1.682 1.734 10.575

10.693

10.841

11.024

11.243

11.499

11.790

12.119

12.484

12.886

13.021

350

OPTIMIZATION IN STATISTICS

of 2.733, 2.944, and 2.513 lbrplot, respectively, results in an estimated maximum yield of snap beans of 13.021 lbrplot. 8.3.3. Modified Ridge Analysis Optimization of ˆ y Žx. on a hypersphere S by the method of ridge analysis is justified provided that the prediction variance on S is relatively small. Furthermore, it is desirable that this variance remain constant on S. If not, then it is possible to obtain poor estimates of the optimum response, especially when the dispersion in the prediction variances on S is large. Thus the reliability of ridge analysis as an optimum-seeking procedure depends very much on controlling the size and variability of the prediction variance. If the design is rotatable, then the prediction variance, Varw ˆ y Žx.x, is constant on S. It is then easy to attain small prediction variances by restricting the procedure to hyperspheres of small radii. However, if the design is not rotatable, then Varw ˆ y Žx.x may vary widely on S, which, as was mentioned earlier, can adversely affect the quality of estimation of the optimum response. This suggests that the prediction variance should be given serious consideration in the strategy of ridge analysis if the design used is not rotatable. Khuri and Myers Ž1979. proposed a certain modification to the method of ridge analysis: one that optimizes ˆ y Žx. subject to a particular constraint on the prediction variance. The following is a description of their proposed modification: Consider model Ž8.12., which can be written as y Ž x . s f⬘ Ž x . ␥ q ⑀ ,

Ž 8.25 .

where f Ž x . s Ž 1, x 1 , x 2 , . . . , x k , x 1 x 2 , x 1 x 3 , . . . , x ky1 x k , x 12 , x 22 , . . . , x k2 . ⬘, ␥ s Ž ␤ 0 , ␤ 1 , ␤ 2 , . . . , ␤ k , ␤ 12 , ␤ 13 , . . . , ␤ ky1 , k , ␤ 11 , ␤ 22 , . . . , ␤ k k . ⬘. The predicted response is given by

ˆy Ž x . s f⬘ Ž x . ␥ˆ ,

Ž 8.26 .

where ␥ ˆ is the least-squares estimator of ␥, namely, ␥ ˆ s Ž X⬘X .

y1

X⬘y,

Ž 8.27 .

where X s wf Žx 1 ., f Žx 2 ., . . . , f Žx n .x⬘ with x i being the vector of design settings at the ith experimental run Ž i s 1, 2, . . . , n, where n is the number of runs used in the experiment., and y is the corresponding vector of n observations. Since Var Ž ␥ ˆ . s Ž X⬘X .

y1

␴ 2,

Ž 8.28 .

351

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

where ␴ 2 is the error variance, then from equation Ž8.26., the prediction variance is of the form Var ˆ y Ž x . s ␴ 2 f⬘ Ž x . Ž X⬘X .

y1

f Ž x. .

Ž 8.29 .

The number of unknown parameters in model Ž8.25. is ps Ž k q 1.Ž k q 2.r2, where k is the number of input variables. Let ␯ 1 , ␯ 2 , . . . , ␯p denote the eigenvalues of X⬘X. Then from equation Ž8.29. and Theorem 2.3.16 we have

␴ 2 f⬘ Ž x . f Ž x . ␯max

F Var ˆ y Ž x. F

␴ 2 f⬘ Ž x . f Ž x . ␯min

,

where ␯min and ␯max are, respectively, the smallest and largest of the ␯ i ’s. This double inequality shows that the prediction variance can be inflated if X⬘X has small eigenvalues. This occurs when the columns of X are multicollinear Žsee, for example, Myers, 1990, pages 125᎐126 and Chapter 8.. Now, by the spectral decomposition theorem ŽTheorem 2.3.10., X⬘X s V⌳V⬘, where V is an orthogonal matrix of orthonormal eigenvectors of X⬘X and ⌳ s DiagŽ ␯ 1 , ␯ 2 , . . . , ␯p . is a diagonal matrix of eigenvalues of X⬘X. Equation Ž8.29. can then be written as p

Var ˆ y Ž x. s ␴

2

Ý

f⬘ Ž x . vj

␯j

js1

2

Ž 8.30 .

,

where vj is the jth column of V Ž j s 1, 2, . . . , p .. If we denote the elements of vj by ®0 j , ®1 j , . . . , ®k j , ®12 j , ®13 j , . . . , ®ky1 k j , ®11 j , ®22 j , . . . , ®k k j , then f⬘Žx.vj can be expressed as f⬘ Ž x . vj s ®0 j q x⬘␶ j q x⬘Tj x,

j s 1, 2, . . . , p,

Ž 8.31 .

where ␶ j s Ž ®1 j , ®2 j , . . . , ®k j .⬘ and ®11 j

1 2

®12 j

®22 j

1 2

®13 j

⭈⭈⭈

1 2

®1 k j

1 2

®23 j

⭈⭈⭈ .. .

1 2

®2 k j . . .

.. .

Tj s

.. .

1 2

j s 1, 2, . . . , p.

,

®ky1 k j ®k k j

symmetric

We note that the form of f⬘Žx.vj , as given by formula Ž8.31., is identical to that of a second-order model. Formula Ž8.30. can then be written as p

Var ˆ y Ž x. s ␴ 2

Ý

js1

Ž ®0 j q x⬘␶ j q x⬘Tj x . ␯j

2

.

Ž 8.32 .

352

OPTIMIZATION IN STATISTICS

y Žx. to have As was noted earlier, small values of ␯ j Ž j s 1, 2, . . . , p . cause ˆ large variances. To reduce the size of the prediction variance within the region explored by ridge analysis, we can consider putting constraints on the portion of Varw ˆ y Žx.x y Žx. subject to the that corresponds to ␯min . It makes sense to optimize ˆ constraints x⬘x s r 2 , ®0 m q x⬘␶m q x⬘Tm x F q,

Ž 8.33 . Ž 8.34 .

where ®0 m , ␶m , and Tm are the values of ®0 , ␶, and T that correspond to ␯min . Here, q is a positive constant chosen small enough to offset the small value of ␯min . Khuri and Myers Ž1979. suggested that q be equal to the largest value taken by < ®0 m q x⬘␶m q x⬘Tm x < at the n design points. The rationale behind this rule of thumb is that the prediction variance is smaller at the design points than at other points in the experimental region. The modification suggested by Khuri and Myers Ž1979. amounts to adding the constraint Ž8.34. to the usual procedure of ridge analysis. In this way, some control can be maintained on the size of prediction variance during the optimization process. The mathematical algorithm needed for this constrained optimization is based on a technique introduced by Myers and Carter Ž1973. for a dual response system in which a primary second-order response function is optimized subject to the condition that a constrained second-order response function takes on some specified or desirable values. Here, the primary response is ˆ y Žx. and the constrained response is ®0 m q x⬘␶m q x⬘Tm x. Myers and Carter’s Ž1973. procedure is based on the method of Lagrange multipliers, which uses the function

ˆ q x⬘Bx ˆ y ␮ Ž ®0 m q x⬘␶m q x⬘Tm x y ␻ . y ␭Ž x⬘x y r 2 . , L s ␤ˆ0 q x⬘␤ ˆ and B ˆ are the same as in model Ž8.14., ␮ and ␭ are Lagrange where ␤ˆ0 , ␤, multipliers, ␻ is such that < ␻ < F q wsee inequality Ž8.34.x, and r is the radius of a hypersphere centered at the origin and contained inside a region of interest R. By differentiating L with respect to x 1 , x 2 , . . . , x k and equating the derivatives to zero, we obtain

Ž Bˆ y ␮Tm y ␭I k . x s Ž ␮ ␶m y ␤ˆ . . 1 2

Ž 8.35 .

As in the method of ridge analysis, to solve equation Ž8.35., values of ␮ and ␭ are chosen directly in such a way that the solution represents a point of maximum Žor minimum. for ˆ y Žx.. Thus for a given value of ␮ , the matrix of ˆ y ␮ Tm y ␭I k ., is made second-order partial derivatives of L, namely 2ŽB w Ž . negative definite and hence a maximum of ˆ y x is achievedx by selecting ␭

OPTIMIZATION TECHNIQUES IN RESPONSE SURFACE METHODOLOGY

353

ˆ y ␮ Tm . Values of ␭ smaller than the larger than the largest eigenvalue of B ˆ y ␮ Tm should be considered in order for ˆy Žx. to smallest eigenvalue of B attain a minimum. It follows that for such an assignment of values for ␮ and ␭, the corresponding solution of equation Ž8.35. produces an optimum for ˆ y subject to a fixed r s Žx⬘x.1r2 and a fixed value of ®0 m q x⬘␶m q x⬘Tm x. EXAMPLE 8.3.2. An attempt was made to design an experiment from which one could find conditions on concentration of three basic substances that maximize a certain mechanical modular property of a solid propellant. The initial intent was to construct and use a central composite design Žsee Khuri and Cornell, 1996, Section 4.5.3. for the three components in the system. However, certain experimental difficulties prohibited the use of the design as planned, and the design used led to problems with multicollinearity as far as the fitting of the second-order model is concerned. The design settings and corresponding response values are given in Table 8.3. In this example, the smallest eigenvalue of X⬘X is ␯min s 0.0321. Correspondingly, the values of ®0 m , ␶m , and Tm in inequality Ž8.34. are ®0 m s y0.2935, ␶m s Ž0.0469, 0.4081, 0.4071.⬘, and 0.1129 Tm s 0.0095 0.2709

0.0095 y0.1382 y0.0148

0.2709 y0.0148 . 0.6453

As for q in inequality Ž8.34., values of < ®0 m q x⬘␶m q x⬘Tm x < were computed at each of the 15 design points in Table 8.3. The largest value was found to

Table 8.3. Design Settings and Response Values for Example 8.3.2 x1

x2

x3

y

y 1.020 0.900 0.870 y0.950 y0.930 0.750 0.830 y0.950 1.950 y2.150 y0.550 y0.450 0.150 0.100 1.450

y1.402 0.478 y1.282 0.458 y1.242 0.498 y1.092 0.378 y0.462 y0.402 0.058 1.378 1.208 1.768 y0.342

y0.998 y0.818 0.882 0.972 y0.868 y0.618 0.732 0.832 0.002 y0.038 y0.518 0.182 0.082 y0.008 0.182

13.5977 12.7838 16.2780 14.1678 9.2461 17.0167 13.4253 16.0967 14.5438 20.9534 11.0411 21.2088 25.5514 33.3793 15.4341

Source: Khuri and Myers Ž1979.. Reproduced with permission of the American Statistical Association.

354

OPTIMIZATION IN STATISTICS

Table 8.4. Results of Modified Ridge Analysis r 0.848 1.162 1.530 1.623 1.795 1.850 1.904 1.935 2.000 < ␻< 0.006 0.074 0.136 1.139 0.048 0.086 0.126 0.146 0.165 5.336 Varw ˆ y Žx.xr␴ 2 1.170 1.635 2.922 3.147 1.305 2.330 3.510 4.177 x1 0.410 0.563 0.773 0.785 0.405 0.601 0.750 0.820 0.965 x2 0.737 1.015 1.320 1.422 1.752 1.751 1.750 1.752 1.752 x3 0.097 0.063 0.019 0.011 0.000 0.000 0.012 0.015 y0.030 22.420 27.780 35.242 37.190 37.042 40.222 42.830 44.110 46.260 ˆy Žx. Source: Khuri and Myers Ž1979.. Reproduced with permission of the American Statistical Association.

Table 8.5. Results of Standard Ridge Analysis r < ␻<

␻ 2r␯ min Var w ˆ y Žx .xr␴ x1 x2 x3 ˆy Žx .

2

0.140 0.241 1.804 2.592 0.037 0.103 0.087

0.379 0.124 0.477 1.554 0.152 0.255 0.235

0.698 0.104 0.337 2.104 0.352 0.422 0.431

0.938 1.146 1.394 1.484 1.744 1.944 1.975 2.000 0.337 0.587 0.942 1.085 1.553 1.958 2.025 2.080 3.543 10.718 27.631 36.641 75.163 119.371 127.815 134.735 6.138 14.305 32.787 42.475 83.38 129.834 138.668 145.907 0.515 0.660 0.835 0.899 1.085 1.227 1.249 1.265 0.531 0.618 0.716 0.749 0.845 0.916 0.927 0.936 0.577 0.705 0.858 0.912 1.074 1.197 1.217 1.232

12.796 16.021 21.365 26.229 31.086 37.640 40.197 48.332

55.147

56.272

57.176

Source: Khuri and Myers Ž1979.. Reproduced with permission of the American Statistical Association.

be 0.087. Hence, the value of < ®0 m q x⬘␶m q x⬘Tm x < should not grow much larger than 0.09 in the experimental region. Furthermore, r in equation Ž8.33. must not exceed the value 2, since most of the design points are contained inside a sphere of radius 2. The results of maximizing ˆ y Žx. subject to this dual constraint are given in Table 8.4. For the sake of comparison, the results of applying the standard procedure of ridge analysis Žthat is, without the additional constraint concerning ®0 m q x⬘␶m q x⬘Tm x. are displayed in Table 8.5. It is clear from Tables 8.4 and 8.5 that the extra constraint concerning ®0 m q x⬘␶m q x⬘Tm x has profoundly improved the precision of ˆ y at the estimated maxima. At a specified radius, the value of ˆ y obtained under standard ridge analysis is higher than the one obtained under modified ridge analysis. However, the prediction variance values under the latter procedure are much smaller, as can be seen from comparing Tables 8.4 and 8.5. While the tradeoff that exists between a high response value and a small prediction variance is a bit difficult to cope with from a decision making standpoint, there is a clear superiority of the results displayed in Table 8.4. For example, one would hardly choose any operating conditions in Table 8.5 that indicate ˆy G 50, due to the accompanying large prediction variances. On the other hand, Table 8.4 reveals that at radius r s 2.000, ˆ y s 46.26 with Varw ˆ y Žx.xr␴ 2

355

RESPONSE SURFACE DESIGNS

s 5.336, while a rival set of coordinates at r s 1.744 for standard ridge analysis gives ˆ y s 48.332 with Varw ˆ y Žx.xr␴ 2 s 83.38. Row 3 of Table 8.5 gives values of ␻ 2r␯min , which should be compared with the corresponding values in row 4 of the same table. One can easily see that in this example, ␻ 2r␯min accounts for a large portion of Varw ˆ y Žx.xr␴ 2 . 8.4. RESPONSE SURFACE DESIGNS We recall from Section 8.3 that one of the objectives of response surface methodology is the selection of a response surface design according to a certain optimality criterion. The design selection entails the specification of the settings of a group of input variables that can be used as experimental runs in a given experiment. The proper choice of a response surface design can have a profound effect on the success of a response surface exploration. To see this, let us suppose that the fitted model is linear of the form

Ž 8.36 .

y s X␤ q ⑀ ,

where y is an n = 1 vector of observations, X is an n = p known matrix that depends on the design settings, ␤ is a vector of p unknown parameters, and ⑀ is a vector of random errors in the elements of y. Typically, ⑀ is assumed to have the normal distribution N Ž0, ␴ 2 I n ., where ␴ 2 is unknown. In this case, ˆ which is given by the vector ␤ is estimated by the least-squares estimator ␤,

ˆ s Ž X⬘X . ␤

y1

X⬘y.

Ž 8.37 .

If x 1 , x 2 , . . . , x k are the input variables for the model under consideration, then the predicted response at a point x s Ž x 1 , x 2 , . . . , x k .⬘ in a region of interest R is written as

ˆ, ˆy Ž x . s f⬘ Ž x . ␤

Ž 8.38 .

where f Žx. is a p= 1 vector whose first element is equal to one and whose remaining py 1 elements are functions of x 1 , x 2 , . . . , x k . These functions are in the form of powers and cross products of powers of the x i ’s up to degree d. In this case, the model is said to be of order d. At the uth experimental run, x u s Ž x u1 , x u1 , . . . , x u k .⬘ and the corresponding response value is yuŽ u s 1, 2, . . . , n.. Then n = k matrix D s wx 1: x 2 : ⭈⭈⭈ :x n x⬘ is the design matrix. Thus by a choice of design we mean the specification of the elements of D. ˆ is an unbiased estimator of ␤ and its If model Ž8.36. is correct, then ␤ variance᎐covariance matrix is given by

ˆ . s Ž X⬘X . Var Ž ␤

y1

␴ 2.

Ž 8.39 .

356

OPTIMIZATION IN STATISTICS

Hence, from formula Ž8.38., the prediction variance can be written as Var ˆ y Ž x . s ␴ 2 f⬘ Ž x . Ž X⬘X .

y1

f Ž x. .

Ž 8.40 .

The design D is rotatable if Varw ˆ y Žx.x remains constant at all points that are equidistant from the design center, as we may recall from Section 8.3. The input variables are coded so that the center of the design coincides with the origin of the coordinates system Žsee Khuri and Cornell, 1996, Section 2.8.. 8.4.1. First-Order Designs If model Ž8.36. is of the first order Žthat is, ds 1., then the matrix X is of the form X s w1 n : Dx, where 1 n is a vector of ones of order n = 1 wsee model Ž8.8.x. The input variables can be coded in such a way that the sum of the elements in each column of D is equal to zero. Consequently, the prediction variance in formula Ž8.40. can be written as Var ˆ y Ž x. s ␴ 2

1 n

q x⬘ Ž D⬘D .

y1

x .

Ž 8.41 .

Formula Ž8.41. clearly shows the dependence of the prediction variance on the design matrix. A reasonable criterion for the choice of D is the minimization of Varw ˆ y Žx.x, or equivalently, the minimization of x⬘ŽD⬘D.y1 x within the region R. To accomplish this we first note that for any x in the region R, x⬘ Ž D⬘D .

y1

x F 5 x 5 22 5 Ž D⬘D .

y1

52 ,

Ž 8.42 .

where 5 x 5 2 s Žx⬘x.1r2 and 5ŽD⬘D.y1 5 2 s wÝ kis1 Ý kjs1 Ž d i j . 2 x1r2 is the Euclidean norm of ŽD⬘D.y1 with d i j being its Ž i, j .th element Ž i, j s 1, 2, . . . , k .. Inequality Ž8.42. follows from applying Theorems 2.3.16 and 2.3.20. Thus by choosing the design D so that it minimizes 5ŽD⬘D.y1 5 2 , the quantity x⬘ŽD⬘D.y1 x, and hence the prediction variance, can be reduced throughout the region R. Theorem 8.4.1. For a given number n of experimental runs, 5ŽD⬘D.y1 5 2 attains its minimum if the columns d 1 , d 2 , . . . , d k of D are such that dXi d j s 0, i / j, and dXi d i is as large as possible inside the region R. Proof. We have that D s wd 1 : d 2 : ⭈⭈⭈ : d k x. The elements of d, are the n design settings of the input variable x i Ž i s 1, 2, . . . , k .. Suppose that the region R places the following restrictions on these settings: dXi d i F c i2 ,

i s 1, 2, . . . , k,

Ž 8.43 .

357

RESPONSE SURFACE DESIGNS

where c i is some fixed constant. This means that the spread of the design in the direction of the ith coordinate axis is bounded by c i2 Ž i s 1, 2, . . . , k .. I Now, if d ii denotes the ith diagonal element of D⬘D, then d ii s dXi d i Ž i s 1, 2, . . . , k .. Furthermore, d ii G

1 dii

Ž 8.44 .

i s 1, 2, . . . , k,

,

where d ii is the ith diagonal element of ŽD⬘D.y1 . To prove inequality Ž8.44., let Di be a matrix of order n = Ž k y 1. obtained from D by removing its ith column d i Ž i s 1, 2, . . . , k .. The cofactor of d ii in D⬘D is then detŽDXi Di .. Hence, from Section 2.3.3, d s ii

det Ž DXi Di . det Ž D⬘D .

Ž 8.45 .

i s 1, 2, . . . , k.

,

There exists an orthogonal matrix E i of order k = k Žwhose determinant has an absolute value of one. such that the first column of DE i is d i and the remaining columns are the same as those of Di , that is, DE i s w d i : Di x ,

i s 1, 2, . . . , k.

It follows that wsee property 7 in Section 2.3.3x det Ž D⬘D . s det Ž EXi D⬘DE i . s det Ž DXi Di . dXi d i y dXi Di Ž DXi Di .

y1

DXi d i .

Hence, from Ž8.45. we obtain d ii s d ii y dXi Di Ž DXi Di .

y1

DXi d i

y1

,

i s 1, 2, . . . , k.

Ž 8.46 .

Inequality Ž8.44. now follows from formula Ž8.46., since dXi Di ŽDXi Di .y1 DXi d i G 0. We can therefore write

Ž D⬘D .

y1

k

2G

1r2

Ý Žd .

is1

ii 2

k

G

Ý

is1

1 d ii2

1r2

.

358

OPTIMIZATION IN STATISTICS

Using the restrictions Ž8.43. we then have

Ž D⬘D .

y1

k

2G

Ý

is1

1 c i4

1r2

.

Equality is achieved if the columns of D are orthogonal to one another and d i i s c i2 Ž i s 1, 2, . . . , k .. This follows from the fact that d i i s 1rd i i if and only if dXi Di s 0 Ž i s 1, 2, . . . , k ., as can be seen from formula Ž8.46.. Definition 8.4.1. A design for fitting a fist-order model is said to be orthogonal if its columns are orthogonal to one another. I Corollary 8.4.1. For a given number n of experimental runs, VarŽ ␤ˆi . attains a minimum if and only if the design is orthogonal, where ␤ˆi is the least-squares estimator of ␤i in model Ž8.8., i s 1, 2, . . . , k. Proof. This follows directly from Theorem 8.4.1 and the fact that VarŽ ␤ˆi . I s ␴ 2 d ii Ž i s 1, 2, . . . , k ., as can be seen from formula Ž8.39.. From Theorem 8.4.1 and Corollary 8.4.1 we conclude that an orthogonal design for fitting a first-order model has optimal variance properties. Another advantage of orthogonal first-order designs is that the effects of the k input variables in model Ž8.8., as measured by the values of the ␤i ’s Ž i s 1, 2, . . . , k ., can be estimated independently. This is because the off-diagonal ˆ in formula Ž8.39. are zero. elements of the variance᎐covariance matrix of ␤ ˆ This means that the elements of ␤ are uncorrelated and hence statistically independent under the assumption of normality of the random error vector ⑀ in model Ž8.36.. Examples of first-order orthogonal designs are given in Khuri and Cornell Ž1996, Chapter 3.. Prominent among these designs are the 2 k factorial design Žeach input variable has two levels, and the number of all possible combinations of these levels is 2 k . and the Plackett᎐Burman design, which was introduced in Plackett and Burman Ž1946.. In the latter design, the number of design points is equal to k q 1, which must be a multiple of 4. 8.4.2. Second-Order Designs These designs are used to fit second-order models of the form given by Ž8.12.. Since the number of parameters in this model is ps Ž k q 1.Ž k q 2.r2, the number of experimental runs Žor design points. in a second-order design must at least be equal to p. The most frequently used second-order designs include the 3 k design Žeach input variable has three levels, and the number of all possible combinations of these levels is 3 k ., the central composite design ŽCCD., and the Box᎐Behnken design.

359

RESPONSE SURFACE DESIGNS

The CCD was introduced by Box and Wilson Ž1951.. It is made up of a factorial portion consisting of a 2 k factorial design, an axial portion of k pairs of points with the ith pair consisting of two symmetric points on the ith coordinate axis Ž i s 1, 2, . . . , k . at a distance of ␣ Ž) 0. from the design center Žwhich coincides with the center of the coordinates system by the coding scheme., and n 0 ŽG 1. center-point runs. The values of ␣ and n 0 can be chosen so that the CCD acquires certain desirable features Žsee, for example, Khuri and Cornell, 1996, Section 4.5.3.. In particular, if ␣ s F 1r4 , where F denotes the number of points in the factorial portion, then the CCD is rotatable. The choice of n 0 can affect the stability of the prediction variance. The Box᎐Behnken design, introduced in Box and Behnken Ž1960., is a subset of a 3 k factorial design and, in general, requires many fewer points. It also compares favorably with the CCD. A thorough description of this design is given in Box and Draper Ž1987, Section 15.4.. Other examples of second-order designs are given in Khuri and Cornell Ž1996, Chapter 4.. 8.4.3. Variance and Bias Design Criteria We have seen that the minimization of the prediction variance represents an important criterion for the selection of a response surface design. This criterion, however, presumes that the fitted model is correct. There are many situations in which bias in the predicted response can occur due to fitting the wrong model. We refer to this as model bias. Box and Draper Ž1959, 1963. presented convincing arguments in favor of recognizing bias as an important design criterionᎏin certain cases, even more important than the variance criterion. Consider again model Ž8.36.. The response value at a point x s Ž x 1 , x 2 , . . . , x k .⬘ in a region R is represented as y Ž x . s f⬘ Ž x . ␤ q ⑀ ,

Ž 8.47 .

where f⬘Žx. is the same as in model Ž8.38.. While it is hoped that model Ž8.47. is correct, there is always a fear that the true model is different. Let us therefore suppose that in reality the true mean response at x, denoted by ␩ Žx., is given by

␩ Ž x . s f⬘ Ž x . ␤ q g⬘ Ž x . ␦

Ž 8.48 .

where the elements of g⬘Žx. depend on x and consist of powers and cross products of powers of x 1 , x 2 , . . . , x k of degree d⬘ ) d, with d being the order of model Ž8.47., and ␦ is a vector of q unknown parameters. For a given

360

OPTIMIZATION IN STATISTICS

design D of n experimental runs, we then have the model ␩ s X␤ q Z␦ , where ␩ is the vector of true means Žor expected values. of the elements of y at the n design points, X is the same as in model Ž8.36., and Z is a matrix of order n = q whose uth row is equal to g⬘Žx u .. Here, x Xu denotes the uth row of D Ž u s 1, 2, . . . , n.. At each point x in R, the mean squared error ŽMSE. of ˆ y Žx., where ˆ y Žx. is Ž . the predicted response as given by formula 8.38 , is defined as MSE ˆ y Ž x. s E ˆ y Ž x. y ␩ Ž x. . 2

This can be expressed as MSE ˆ y Ž x . s Var ˆ y Ž x . q Bias 2 ˆ y Ž x. ,

Ž 8.49 .

where Biasw ˆ y Žx. s E w ˆ y Žx.x y ␩ Žx.. The fundamental philosophy of Box and Draper Ž1959, 1963. is centered around the consideration of the integrated mean squared error ŽIMSE. of ˆ y Žx.. This is denoted by J and is defined in terms of a k-tuple Riemann integral over the region R, namely, Js

n⍀

␴2

HRMSE

ˆy Ž x . dx,

Ž 8.50 .

where ⍀y1 s HR dx and ␴ 2 is the error variance. The partitioning of MSEw ˆ y Žx.x as in formula Ž8.49. enables us to separate J into two parts: Js

n⍀

␴

2

HRVar

ˆy Ž x . dx q

n⍀

HRBias

␴2

2

ˆy Ž x . dx s V q B.

Ž 8.51 .

The quantities V and B are called the average variance and average squared bias of ˆ y Žx., respectively. Both V and B depend on the design D. Thus a reasonable choice of design is one that minimizes Ž1. V alone, Ž2. B alone, or Ž3. J s V q B. Now, using formula Ž8.40., V can be written as

HRf⬘ Ž x. Ž X⬘X.

V s n⍀

½

y1

f Ž x . dx

HRf Ž x. f⬘ Ž x. dx

s tr n Ž X⬘X .

y1

⍀

s tr n Ž X⬘X .

y1

⌫11 ,

5 Ž 8.52 .

361

RESPONSE SURFACE DESIGNS

where

HRf Ž x. f⬘ Ž x. dx.

⌫11 s ⍀

As for B, we note from formula Ž8.37. that

ˆ . s Ž X⬘X . E Ž␤

y1

X⬘␩

s␤qA␦, where A s ŽX⬘X.y1 X⬘Z. Thus from formula Ž8.38. we have E ˆ y Ž x . s f⬘ Ž x . Ž ␤ q A ␦ . . Using the expression for ␩ Žx. in formula Ž8.48., B can be written as Bs s s s s

n⍀

␴2 n⍀

␴2 n⍀

␴2 n⍀

␴2 n ␴2

HR f⬘ Ž x. ␤ q f⬘ Ž x. A ␦ y f⬘ Ž x. ␤ y g⬘Ž x. ␦ HR f⬘ Ž x. A ␦ y g⬘Ž x. ␦ HR␦⬘ A⬘f Ž x. y gŽ x.

2

2

dx

dx

f⬘ Ž x . A y g⬘ Ž x . ␦ dx

HR␦⬘ A⬘f Ž x. f⬘ Ž x. A y gŽ x. f⬘ Ž x. A y A⬘f Ž x. g⬘ Ž x. q g Ž x. g⬘ Ž x.

␦ dx

␦⬘⌬ ␦ ,

Ž 8.53 .

where X ⌬ s A⬘⌫11 A y ⌫12 A y A⬘⌫12 q ⌫22 ,

HRf Ž x. g⬘Ž x. dx,

⌫12 s ⍀

HRgŽ x. g⬘Ž x. dx.

⌫22 s ⍀

The matrices ⌫11 , ⌫12 , and ⌫22 are called region moments. By adding and X y1 subtracting the matrix ⌫12 ⌫11 ⌫12 from ⌬ in formula Ž8.53., B can be expressed as Bs

n

␴2

X y1 y1 y1 ␦⬘ Ž ⌫22 y ⌫12 ⌫11 ⌫12 . q Ž A y ⌫11 ⌫12 . ⬘⌫11 Ž A y ⌫11 ⌫12 . ␦. Ž 8.54 .

362

OPTIMIZATION IN STATISTICS

We note that the design D affects only the second expression inside brackets on the right-hand side of formula Ž8.54.. Thus to minimize B, the design D should be chosen such that y1 A y ⌫11 ⌫12 s 0.

Ž 8.55 .

Since A s ŽX⬘X.y1 X⬘Z, a sufficient Žbut not necessary . condition for the minimization of B is M 11 s ⌫11 ,

M 12 s ⌫12 ,

Ž 8.56 .

where M 11 s Ž1rn.X⬘X, M 12 s Ž1rn.X⬘Z are the so-called design moments. Thus a sufficient condition for the minimization of B is the equality of the design moments, M 11 and M 12 , to the corresponding region moments, ⌫11 and ⌫12 . The minimization of J s V q B is not possible without the specification of ␦r␴ . Box and Draper Ž1959, 1963. showed that unless V is considerably larger than B, the optimal design that minimizes J has characteristics similar to those of a design that minimizes just B. Examples of designs that minimize V alone or B alone can be found in Box and Draper Ž1987, Chapter 13., Khuri and Cornell Ž1996, Chapter 6., and Myers Ž1976, Chapter 9..

8.5. ALPHABETIC OPTIMALITY OF DESIGNS Let us again consider model Ž8.47., which we now assume to be correct, that is, the true mean response, ␩ Žx., is equal to f⬘Žx.␤. In this case, the matrix X⬘X plays an important role in the determination of an optimal design, since the elements of ŽX⬘X.y1 are proportional to the variances and covariances of the least-squares estimators of the model’s parameters wsee formula Ž8.39.x. The mathematical theory of optimal designs, which was developed by Kiefer Ž1958, 1959, 1960, 1961, 1962a, b., is concerned with the choice of designs that minimize certain functions of the elements of ŽX⬘X.y1 . The kernel of Kiefer’s approach is based on the concept of design measure, which represents a generalization of the traditional design concept. So far, each of the designs that we have considered for fitting a response surface model has consisted of a set of n points in a k-dimensional space Ž k G 1.. Suppose that x 1 , x 2 , . . . , x m are distinct points of an n-point design Ž m F n. with the lth Ž l s 1, 2, . . . , m. point being replicated n l ŽG 1. times Žthat is, n l repeated observations are taken at this point.. The design can therefore be regarded as a collection of points in a region of interest R with the lth point being assigned the weight n lrn Ž l s 1, 2, . . . , m., where n s Ý m ls1 n l . Kiefer generalized this setup using the so-called continuous design measure, which is

363

ALPHABETIC OPTIMALITY OF DESIGNS

basically a probability measure ␰ Žx. defined on R and satisfies the conditions

␰ Ž x . G 0 for all x g R and

HRd ␰ Ž x. s 1.

In particular, the measure induced by a traditional design D with n points is called a discrete design measure and is denoted by ␰ n . It should be noted that while a discrete design measure is realizable in practice, the same is not true of a general continuous design measure. For this reason, the former design is called exact and the latter design is called approximate. By definition, the moment matrix of a design measure ␰ is a symmetric matrix of the form MŽ ␰ . s w m i j Ž ␰ .x, where mi j Ž ␰ . s

HR f Ž x. f Ž x. d ␰ Ž x. . i

j

Ž 8.57 .

Here, f i Žx. is the ith element of f Žx. in formula Ž8.47., i s 1, 2, . . . , p. For a discrete design measure ␰ n , the Ž i, j .th element of the moment matrix is mi j Ž ␰n . s

1 n

m

Ý n l fi Ž x l . f j Ž x l . ,

Ž 8.58 .

ls1

where m is the number of distinct design points and n l is the number of replications at the lth point Ž l s 1, 2, . . . , m.. In this special case, the matrix MŽ ␰ . reduces to the usual moment matrix Ž1rn.X⬘X, where X is the same matrix as in formula Ž8.36.. For a general design measure ␰ , the standardized prediction variance, denoted by dŽx, ␰ ., is defined as d Ž x, ␰ . s f⬘ Ž x . M Ž ␰ .

y1

f Ž x. ,

Ž 8.59 .

where MŽ ␰ . is assumed to be nonsingular. In particular, for a discrete design measure ␰ n , the prediction variance in formula Ž8.40. is equal to Ž ␴ 2rn. dŽx, ␰ n .. Let H denote the class of all design measures defined on the region R. A prominent design criterion that has received a great deal of attention is that of D-optimality, in which the determinant of MŽ ␰ . is maximized. Thus a design measure ␰ d is D-optimal if det M Ž ␰ d . s sup det M Ž ␰ . . ␰gH

Ž 8.60 .

The rationale behind this criterion has to do with the minimization of the ˆ of the parameter vector generalized variance of the least-squares estimator ␤ ˆ ␤. By definition, the generalized variance of ␤ is the same as the determiˆ This is based on the fact that nant of the variance᎐covariance matrix of ␤.

364

OPTIMIZATION IN STATISTICS

under the normality assumption, the content Žvolume. of a fixed-level confidence region on ␤ is proportional to wdetŽX⬘X.xy1r2 . The review articles by St. John and Draper Ž1975., Ash and Hedayat Ž1978., and Atkinson Ž1982, 1988. contain many references on D-optimality. Another design criterion that is closely related to D-optimality is Goptimality, which is concerned with the prediction variance. By definition, a design measure ␰ g is G-optimal if it minimizes over H the maximum standardized prediction variance over the region R, that is, sup d Ž x, ␰ g . s inf

␨gH

xgR

½ sup d Ž x, ␨ . 5 .

Ž 8.61 .

xgR

Kiefer and Wolfowitz Ž1960. showed that D-optimality and G-optimality, as defined by formulas Ž8.60. and Ž8.61., are equivalent. Furthermore, a design measure ␰ * is G-optimal Žor D-optimal. if and only if sup d Ž x, ␰ * . s p,

Ž 8.62 .

xgR

where p is the number of parameters in the model. Formula Ž8.62. can be conveniently used to determine if a given design measure is D-optimal, since in general sup x g R dŽx, ␰ . G p for any design measure ␰ g H. If equality can be achieved by a design measure, then it must be G-optimal, and hence D-optimal. EXAMPLE 8.5.1. Consider fitting a second-order model in one input variable x over the region R s wy1, 1x. In this case, model Ž8.47. takes the form y Ž x . s ␤ 0 q ␤ 1 xq ␤ 11 x 2 q ⑀ , that is, f⬘Ž x . s Ž1, x, x 2 .. Suppose that the design measure used is defined as

␰ Ž x. s

½

1 3

xs y1, 0, 1,

,

0

otherwise.

Ž 8.63 .

Thus ␰ is a discrete design measure that assigns one-third of the experimental runs to each of the points y1, 0, and 1. This design measure is D-optimal. To verify this claim, we first need to determine the values of the elements of the moment matrix MŽ ␰ .. Using formula Ž8.58. with n lrns 13 for l s 1, 2, 3, we find that m11 s 1, m12 s 0, m13 s 23 , m 22 s 23 , m 23 s 0, and m 33 s 32 . Hence, 1

0

2 3

MŽ ␰ . s 0

2 3

0 ,

2 3

0

2 3

y1

M

Ž␰ .s

y3

3

0

0

3 2

0 .

y3

0

9 2

365

ALPHABETIC OPTIMALITY OF DESIGNS

By applying formula Ž8.59. we find that dŽ x, ␰ . s 3 y 92 x 2 q 92 x 4 ,y 1 F x F 1. We note that dŽ x, ␰ . F 3 for all x in wy1, 1x with dŽ0, ␰ . s 3. Thus sup x g R dŽ x, ␰ . s 3. Since 3 is the number of parameters in the model, then by condition Ž8.62. we conclude that the design measure defined by formula Ž8.63. is D-optimal. In addition to the D- and G-optimality criteria, other variance-related design criteria have also been investigated. These include A- and E-optimality. By definition, a design measure is A-optimal if it maximizes the trace of MŽ ␰ .. This is equivalent to minimizing the sum of the variances of the least-squares estimators of the fitted model’s parameters. In E-optimality, the smallest eigenvalue of MŽ ␰ . is maximized. The rationale behind this criterion is based on the fact that d Ž x, ␰ . F

f⬘ Ž x . f Ž x .

␭min

,

as can be seen from formula Ž8.59., where ␭min is the smallest eigenvalue of MŽ ␰ .. Hence, dŽx, ␰ . can be reduced by maximizing ␭ min . The efficiency of a design measure ␨ g H with respect to a D-optimal design is defined as

D-efficiency s

½

5

det M Ž ␨ . sup␰ g H det M Ž ␰ .

1rp

,

where p is the number of parameters in the model. Similarly, the Gefficiency of ␨ is defined as G-efficiency s

p sup x g R d Ž x, ␨ .

.

Both D- and G-efficiency values fall within the interval w0, 1x. The closer these values are to 1, the more efficient their corresponding designs are. Lucas Ž1976. compared several second-order designs Žsuch as central composite and Box᎐Behnken designs. on the basis of their D- and G-efficiency values. The equivalence theorem of Kiefer and Wolfowitz Ž1960. can be applied to construct a D-optimal design using a sequential procedure. This procedure is described in Wynn Ž1970, 1972. and goes as follows: Let Dn 0 denote an initial response surface design with n 0 points, x 1 , x 2 , . . . , x n 0 , for which the matrix X⬘X is nonsingular. A point x n 0q1 is found in the region R such that d Ž x n 0q1 , ␰ n 0 . s sup d Ž x, ␰ n 0 . , xgR

366

OPTIMIZATION IN STATISTICS

where ␰ n 0 is the discrete design measure that represents Dn 0 . By augmenting Dn 0 with x n 0q1 we obtain the design Dn 0q1 . Then, another point x n 0q2 is chosen such that d Ž x n 0q2 , ␰ n 0q1 . s sup d Ž x, ␰ n 0q1 . , xgR

where ␰ n 0q1 is the discrete design measure that represents Dn 0q1 . The point x n 0q2 is then added to Dn 0q1 to obtain the design Dn 0q2 . By continuing this process we obtain a sequence of discrete design measures, namely, ␰ n 0 , ␰ n 0q1 , ␰ n 0q2 , . . . . Wynn Ž1970. showed that this sequence converges to the D-optimal design ␰ d , that is, det M Ž ␰ n 0qn . ™ det M Ž ␰ d . as n ™ ⬁. An example is given in Wynn Ž1970, Section 5. to illustrate this sequential procedure. The four design criteria, A-, D-, E-, and G-optimality, are referred to as alphabetic optimality. More detailed information about these criteria can be found in Atkinson Ž1982, 1988., Fedorov Ž1972., Pazman Ž1986., and Silvey Ž1980.. Recall that to perform an actual experiment, one must use a discrete design. It is possible to find a discrete design measure ␰ n that approximates an optimal design measure. The approximation is good whenever n is large with respect to p Žthe number of parameters in the model.. Note that the equivalence theorem of Kiefer and Wolfowitz Ž1960. applies to general design measures and not necessarily to discrete design measures, that is, D- and G-optimality criteria are not equivalent for the class of discrete design measures. Optimal n-point discrete designs, however, can still be found on the basis of maximizing the determinant of X⬘X, for example. In this case, finding an optimal n-point design requires a search involving nk variables, where k is the number of input variables. Several algorithms have been introduced for this purpose. For example, the DETMAX algorithm by Mitchell Ž1974. is used to maximize detŽX⬘X.. A review of algorithms for constructing optimal discrete designs can be found in Cook and Nachtsheim Ž1980. Žsee also Johnson and Nachtsheim, 1983.. One important criticism of the alphabetic optimality approach is that it is set within a rigid framework governed by a set of assumptions. For example, a specific model for the response function must be assumed as the ‘‘true’’ model. Optimal design measures can be quite sensitive to this assumption. Box Ž1982. presented a critique to this approach. He argued that in a response surface situation, it may not be realistic to assume that a model such as Ž8.47. represents the true response function exactly. Some protection against bias in the model should therefore be considered when choosing a response surface design. On the other hand, Kiefer Ž1975. criticized certain aspects of the preoccupation with bias, pointing out examples in which the variance criterion is compromised for the sake of the bias criterion. It follows

367

DESIGNS FOR NONLINEAR MODELS

that design selection should be guided by more than one single criterion Žsee Kiefer, 1975, page 286; Box, 1982, Section 7.. A reasonable approach is to select compromise designs that are sufficiently good Žbut not necessarily optimal. from the viewpoint of several criteria that are important to the user.

8.6. DESIGNS FOR NONLINEAR MODELS The models we have considered so far in the area of response surface methodology were linear in the parameters; hence the term linear models. There are, however, many experimental situations in which linear models do not adequately represent the true mean response. For example, the growth of an organism is more appropriately depicted by a nonlinear model. By definition, a nonlinear model is one of the form y Ž x . s h Ž x, ␪ . q ⑀ ,

Ž 8.64 .

where x s Ž x 1 , x 2 , . . . , x k .⬘ is a vector of k input variables, ␪ s Ž ␪ 1 , ␪ 2 , . . . , ␪p .⬘ is a vector of p unknown parameters, ⑀ is a random error, and hŽx, ␪ . is a known function, nonlinear in at least one element of ␪. An example of a nonlinear model is h Ž x, ␪ . s

␪1 x ␪2 qx

.

Here, ␪ s Ž ␪ 1 , ␪ 2 .⬘ and ␪ 2 is a nonlinear parameter. This particular model is known as the Michaelis᎐Menten model for enzyme kinetics. It relates the initial velocity of an enzymatic reaction to the substrate concentration x. In contrast to linear models, nonlinear models have not received a great deal of attention in response surface methodology, especially in the design area. The main design criterion for nonlinear models is the D-optimality criterion, which actually applies to a linearized form of the nonlinear model. More specifically, this criterion depends on the assumption that in some neighborhood of a specified value ␪ 0 of ␪, the function hŽx, ␪ . is approximately linear in ␪. In this case, a first-order Taylor’s expansion of hŽx, ␪ . yields the following approximation of hŽx, ␪ .: p

h Ž x, ␪ . f h Ž x, ␪ 0 . q

Ý Ž ␪ i y ␪ i0 .

⭸ h Ž x, ␪ 0 .

is1

⭸␪ i

.

Thus if ␪ is close enough to ␪ 0 , then we have approximately the linear model p

z Ž x. s

Ý ␺i

is1

⭸ h Ž x, ␪ 0 . ⭸␪ i

q⑀ ,

Ž 8.65 .

where z Žx. s y Žx. y hŽx, ␪ 0 ., and ␺ i is the ith element of ␺ s ␪ y ␪ 0 Ž i s 1, 2, . . . , p ..

368

OPTIMIZATION IN STATISTICS

For a given design consisting of n experimental runs, model Ž8.65. can be written in vector form as z s HŽ ␪0 . ␺ q ⑀ ,

Ž 8.66 .

where HŽ ␪ 0 . is an n = p matrix whose Ž u, i .th element is ⭸ hŽx u , ␪ 0 .r⭸␪ i with x u being the vector of design settings for the k input variables at the uth experimental run Ž i s 1, 2, . . . , p; u s 1, 2, . . . , n.. Using the linearized form given by model Ž8.66., a design is chosen to maximize the determinant detwH⬘Ž ␪ 0 .HŽ ␪ 0 .x. This is known as the Box᎐Lucas criterion Žsee Box and Lucas, 1959.. It can be easily seen that a nonlinear design obtained on the basis of the Box᎐Lucas criterion depends on the value of ␪ 0 . This is an undesirable characteristic of nonlinear models, since a design is supposed to be used for estimating the unknown parameter vector ␪. By contrast, designs for linear models are not dependent on the fitted model’s parameters. Several procedures have been proposed for dealing with the problem of design dependence on the parameters of a nonlinear model. These procedures are mentioned in the review article by Myers, Khuri, and Carter Ž1989.. See also Khuri and Cornell Ž1996, Section 10.5.. EXAMPLE 8.6.1. Let us again consider the Michaelis᎐Menten model mentioned earlier. The partial derivatives of hŽ x, ␪ . with respect to ␪ 1 and ␪ 2 are

⭸ h Ž x, ␪ . ⭸␪ 1 ⭸ h Ž x, ␪ . ⭸␪ 2

s

s

x

␪2 qx

,

y␪ 1 x

Ž ␪2 qx.

2

.

Suppose that it is desired to find a two-point design that consists of the settings x 1 and x 2 using the Box᎐Lucas criterion. In this case,

HŽ ␪0 . s

s

⭸ h Ž x1 , ␪0 .

⭸ h Ž x1 , ␪0 .

⭸␪ 1

⭸␪ 2

⭸ h Ž x 2 , ␪0 .

⭸ h Ž x 2 , ␪0 .

⭸␪ 1

⭸␪ 2

x1

y␪ 10 x 1

␪ 20 q x 1

Ž ␪ 20 q x 1 .

x2

y␪ 10 x 2

␪ 20 q x 2

Ž ␪ 20 q x 2 .

2

, 2

369

DESIGNS FOR NONLINEAR MODELS

where ␪ 10 and ␪ 20 are the elements of ␪ 0 . In this example, HŽ ␪ 0 . is a square matrix. Hence, det H⬘ Ž ␪ 0 . H Ž ␪ 0 . s
det H Ž ␪ 0 . s

42

2 ␪ 10 x 12 x 22 Ž x 2 y x 1 .

2

Ž ␪ 20 q x 1 . Ž ␪ 20 q x 2 . 4

4

Ž 8.67 .

.

To determine the maximum of this determinant, let us first equate its partial derivatives with respect to x 1 and x 2 to zero. It can be verified that the solution of the resulting equations Žthat is, the stationary point. falls outside the region of feasible values for x 1 and x 2 Žboth x 1 and x 2 must be nonnegative.. Let us therefore restrict our search for the maximum within the region R s
Ž x 1 , x 2 .< 0 F x 1 F x max , 0 F x 2 F x max 4 , where x max is the maximum allowable substrate concentration. Since the partial derivatives of the determinant in formula Ž8.67. do not vanish in R, then its maximum must be attained on the boundary of R. On x 1 s 0, or x 2 s 0, the value of the determinant is zero. If x 1 s x max , then det H⬘ Ž ␪ 0 . H Ž ␪ 0 . s

2 2 ␪ 10 x max x 22 Ž x 2 y x max .

2

Ž ␪ 20 q x max . Ž ␪ 20 q x 2 . 4

4

.

It can be verified that this function of x 2 has a maximum at the point x 2 s ␪ 20 x max rŽ2 ␪ 20 q x max . with a value given by max

x 1sx max


det

H⬘ Ž ␪ 0 . H Ž ␪ 0 . 4 s

2 6 ␪ 10 x max 2 16␪ 20 Ž ␪ 20 q x max .

6

.

Ž 8.68 .

Similarly, if x 2 s x max , then det H⬘ Ž ␪ 0 . H Ž ␪ 0 . s

2 2 ␪ 10 x max x 12 Ž x max y x 1 .

2

Ž ␪ 20 q x 1 . Ž ␪ 20 q x max . 4

4

,

which attains the same maximum value as in formula Ž8.68. at the point x 1 s ␪ 20 x m ax r Ž2 ␪ 20 q x m ax .. We conclude that the maximum of detwH⬘Ž ␪ 0 .HŽ ␪ 0 .x over the region R is achieved when x 1 s x max and x 2 s ␪ 20 x max rŽ2 ␪ 20 q x max ., or when x 1 s ␪ 20 x max rŽ2 ␪ 20 q x max . and x 2 s x max . We can clearly see in this example the dependence of the design settings on ␪ 2 , but not on ␪ 1. This is attributed to the fact that ␪ 1 appears linearly in the model, but ␪ 2 does not. In this case, the model is said to be partially nonlinear. Its D-optimal design depends only on those parameters that do not appear linearly. More details concerning partially nonlinear models can be found in Khuri and Cornell Ž1996, Section 10.5.3..

370

OPTIMIZATION IN STATISTICS

8.7. MULTIRESPONSE OPTIMIZATION By definition, a multiresponse experiment is one in which a number of responses can be measured for each setting of a group of input variables. For example, in a skim milk extrusion process, the responses, y 1 s percent residual lactose and y 2 s percent ash, are known to depend on the input variables, x 1 s pH level, x 2 s temperature, x 3 s concentration, and x 4 s time Žsee Fichtali, Van De Voort, and Khuri, 1990.. As in single-response experiments, one of the objectives of a multiresponse experiment is the determination of conditions on the input variables that optimize the predicted responses. The definition of an optimum in a multiresponse situation, however, is more complex than in the singleresponse case. The reason for this is that when two or more response variables are considered simultaneously, the meaning of an optimum becomes unclear, since there is no unique way to order the values of a multiresponse function. To overcome this difficulty, Khuri and Conlon Ž1981. introduced a multiresponse optimization technique called the generalized distance approach. The following is an outline of this approach: Let r be the number of responses, and n be the number of experimental runs for all the responses. Suppose that these responses can be represented by the linear models yi s X␤ i q ⑀ i ,

i s 1, 2, . . . , r ,

where yi is a vector of observations on the ith response, X is a known matrix of order n = p and rank p, ␤ i is a vector of p unknown parameters, and ⑀ i is a random error vector associated with the ith response Ž i s 1, 2, . . . , r .. It is assumed that the rows of the error matrix w ⑀ 1: ⑀ 2 : ⭈⭈⭈ : ⑀ r x are statistically independent with each having a zero mean vector and a common variance᎐covariance matrix ⌺. Note that the matrix X is assumed to be the same for all the responses. Let x 1 , x 2 , . . . , x k be input variables that influence the r responses. The predicted response value at a point x s Ž x 1 , x 2 , . . . , x k .⬘ in a region R for the ˆ i , where ␤ ˆ i s ŽX⬘X.y1 X⬘yi is the leastith response is given by ˆ yi Žx. s f⬘Žx.␤ Ž . squares estimator of ␤ i i s 1, 2, . . . , r . Here, f⬘Žx. is of the same form as a row of X, except that it is evaluated at the point x. It follows that Var ˆ yi Ž x . s ␴ii f⬘ Ž x . Ž X⬘X .

y1

f Ž x. ,

i s 1, 2, . . . , r ,

Cov ˆ yi Ž x . , ˆ y j Ž x . s ␴i j f⬘ Ž x . Ž X⬘X .

y1

f Ž x. ,

i / j s 1, 2, . . . , r ,

where ␴i j is the Ž i, j .th element of ⌺. The variance᎐covariance matrix of ˆyŽx. s w ˆy 1Žx., ˆy 2 Žx., . . . , ˆyr Žx.x⬘ is then of the form Var ˆ y Ž x . s f⬘ Ž x . Ž X⬘X .

y1

f Ž x. ⌺ .

371

MULTIRESPONSE OPTIMIZATION

ˆ of ⌺ can be used Since ⌺ is in general unknown, an unbiased estimator, ⌺, instead, where ˆs ⌺

1 nyp

Y⬘ I n y X Ž X⬘X .

y1

X⬘ Y,

ˆ is nonsingular provided that Y is of and Y s wy1: y2 : ⭈⭈⭈ : yr x. The matrix ⌺ rank r F n y p. An estimate of Varwˆ yŽx.x is then given by $

Var ˆ y Ž x . s f⬘ Ž x . Ž X⬘X .

y1

ˆ. f Ž x. ⌺

Ž 8.69 .

yi Žx. optimized individually over the Let ␾ i denote the optimum value of ˆ Ž . Ž region R i s 1, 2, . . . , r . Let ␾ s ␾ 1 , ␾ 2 , . . . , ␾r .⬘. These individual optima do not in general occur at the same location in R. To achieve a compromise optimum, we need to find x that minimizes ␳ wˆ yŽx., ␾ x, where ␳ is some Ž . metric that measures the distance of ˆ y x from ␾. One possible choice for ␳ is the metric $

␳ ˆ yŽ x. , ␾ s ˆ y Ž x . y ␾ ⬘
Var ˆ yŽ x.

4

y1

1r2

ˆy Ž x . y ␾

,

which, by formula Ž8.69., can be written as

␳ ˆ yŽ x. , ␾ s

ˆ y1 ˆy Ž x . y ␾ ˆy Ž x . y ␾ ⬘⌺ y1 f⬘ Ž x . Ž X⬘X . f Ž x .

1r2

.

Ž 8.70 .

yŽx. s ␾, that is, when all the responses We note that ␳ s 0 if and only if ˆ attain their individual optima at the same point; otherwise, ␳ ) 0. Such a point Žif it exists. is called a point of ideal optimum. In general, an ideal optimum rarely exists. In order to have conditions that are as close as possible to an ideal optimum, we need to minimize ␳ over the region R. Let us suppose that the minimum occurs at the point x 0 g R. Then, at x 0 the experimental conditions can be described as being near optimal for each of the r response functions. We therefore refer to x 0 as a point of compromise optimum. Note that the elements of ␾ in formula Ž8.70. are random variables since they are the individual optima of ˆ y 1Žx., ˆ y 2 Žx., . . . , ˆ yr Žx.. If the variation associated with ␾ is large, then the metric ␳ may not accurately measure the deviation of ˆ yŽx. from the true ideal optimum. In this case, some account should be taken of the randomness of ␾ in the development of the metric. To do so, let ␨ s Ž ␨ 1 , ␨ 2 , . . . , ␨r .⬘, where ␨ i is the optimum value of the true mean of the ith response optimized individually over the region R Ž i s 1, 2, . . . , r .. Let D␨ be a confidence region for ␨. For a fixed x g R and whenever ␨ g D␨ , we obviously have

␳ ˆ y Ž x . , ␨ F max ␳ ˆ yŽ x. , ␩ . ␩gD␨

Ž 8.71 .

372

OPTIMIZATION IN STATISTICS

yŽx., ␨ x, The right-hand side of this inequality serves as an upper bound on ␳ wˆ which represents the distance of ˆ yŽx. from the true ideal optimum. It follows that

½

min ␳ ˆ y Ž x . , ␨ F min max ␳ ˆ yŽ x. , ␩ xgR

xgR

␩gD ␨

5.

Ž 8.72 .

The right-hand side of this inequality provides a conservative measure of distance between the compromise and ideal optima. The confidence region D␨ can be determined in a variety of ways. Khuri and Conlon Ž1981. considered a rectangular confidence region of the form

␥ 1 i F ␨i F ␥ 2 i ,

i s 1, 2, . . . , r ,

where

␥ 1 i s ␾ i y g i Ž ␰ i . MSi1r2 t␣ r2 , nyp , ␥ 2 i s ␾ i q g i Ž ␰ i . MSi1r2 t␣ r2 , nyp ,

Ž 8.73 .

where MSi is the error mean square for the ith response, ␰ i is the point at which ˆ yi Žx. attains the individual optimum ␾ i , t␣ r2, nyp is the upper Ž ␣r2. = 100th percentile of the t-distribution with n y p degrees of freedom, and g i Ž ␰ i . is given by g i Ž ␰ i . s f⬘ Ž ␰ i . Ž X⬘X .

y1

f Ž␰i.

1r2

,

i s 1, 2, . . . , r .

Khuri and Conlon Ž1981. showed that such a rectangular confidence region has approximately a confidence coefficient of at least 1 y ␣ *, where ␣ * s 1 y Ž1 y ␣ . r. It should be noted that the evaluation of the right-hand side of inequality Ž8.72. requires that ␳ wˆ yŽx., ␩ x be maximized first with respect to ␩ over D␨ for a given x g R. The maximum value thus obtained, being a function of x, is then minimized over the region R. A computer program for the implementation of this min᎐max procedure is described in Conlon and Khuri Ž1992.. A complete electronic copy of the code, along with examples, can be downloaded from the Internet at ftp:rrftp.stat.ufl.edurpubrmr.tar.Z. Numerical examples that illustrate the application of the generalized distance approach for multiresponse optimization can be found in Khuri and Conlon Ž1981. and Khuri and Cornell Ž1996, Chapter 7..

8.8. MAXIMUM LIKELIHOOD ESTIMATION AND THE EM ALGORITHM We recall from Section 7.11.2 that the maximum likelihood ŽML. estimates of a set of parameters, ␪ 1 , ␪ 2 , . . . , ␪p , for a given distribution maximize the

373

MAXIMUM LIKELIHOOD ESTIMATION AND THE EM ALGORITHM

likelihood function of a sample, X 1 , X 2 , . . . , X n , of size n from the distribution. The ML estimates of the ␪ i ’s denoted by ␪ˆ1 , ␪ˆ2 , . . . , ␪ˆp , can be found by solving the likelihood equations Žthe likelihood function must be differentiable and unimodal.

⭸ log L Ž x, ˆ ␪. ⭸␪ i

s 0,

i s 1, 2, . . . , p,

Ž 8.74 .

where ˆ ␪ s Ž ␪ˆ1 , ␪ˆ2 , . . . , ␪ˆp .⬘, x s Ž x 1 , x 2 , . . . , x n .⬘, and LŽx, ␪ . s f Žx, ␪ . with f Žx, ␪ . being the density function Žor probability mass function. of X s n Ž X 1 , X 2 , . . . , X n .⬘. Note that f Žx, ␪ . can be written as Ł is1 g Ž x i , ␪ ., where g Ž x, ␪ . is the density function Žor probability mass function. associated with the distribution. Equations Ž8.74. may not have a closed-form solution. For example, consider the so-called truncated Poisson distribution whose probability mass function is of the form Žsee Everitt, 1987, page 29. g Ž x, ␪ . s

ey␪␪ x

Ž 1 y ey␪ . x!

xs 1, 2, . . . .

,

Ž 8.75 .

In this case, log L Ž x, ␪ . s log

n

Ł g Ž xi , ␪ .

is1

s yn␪ q Ž log ␪ .

n

n

is1

is1

Ý x i y Ý log x i !y n log Ž 1 y ey␪ . .

Hence,

⭸ L* Ž x, ␪ . ⭸␪

s ynq

1

␪

n

ney␪

Ý x i y 1 y ey␪ ,

Ž 8.76 .

is1

where L*Žx, ␪ . s log LŽx, ␪ . is the log-likelihood function. The likelihood equation, which results from equating the right-hand side of formula Ž8.76. to zero, has no closed-form solution for ␪ . In general, if equations Ž8.74. do not have a closed-form solution, then, as was seen in Section 8.1, iterative methods can be applied to maximize LŽx, ␪ . wor L*Žx, ␪ .x. Using, for example, the Newton᎐Raphson method Žsee Section 8.1.2., if ˆ ␪ 0 is an initial estimate of ␪ and ˆ ␪ i is the estimate at the ith iteration, then by applying formula Ž8.7. we have

ˆ␪ iq1 s ˆ␪ i y Hy1 ˆ ˆ L* Ž x, ␪ i . ⵱L* Ž x, ␪ i . ,

i s 0, 1,2, . . . ,

374

OPTIMIZATION IN STATISTICS

where H L*Žx, ␪ . and ⵱L*Žx, ␪ . are, respectively, the Hessian matrix and gradient vector of the log-likelihood function. Several iterations can be made until a certain convergence criterion is satisfied. A modification of this procedure is the so-called Fisher’s method of scoring, where H L* is replaced by its expected value, that is,

½

5

ˆ␪ iq1 s ˆ␪ i y E H L* Ž x, ˆ␪ i .

y1

⵱L* Ž x, ˆ ␪i . ,

i s 0, 1, 2, . . . . Ž 8.77 .

Here, the expected value is taken with respect to the given distribution. EXAMPLE 8.8.1. ŽEveritt, 1987, pages 30᎐31.. Consider the truncated Poisson distribution described in formula Ž8.75.. In this case, since we only have one parameter ␪ , the gradient takes the form ⵱L*Žx, ␪ . s ⭸ L*Žx, ␪ .r⭸␪ , which is given by formula Ž8.76.. Hence, the Hessian matrix is

⭸ 2 L* Ž x, ␪ .

H L* Ž x, ␪ . s

⭸␪ 2

sy

n

ney␪

is1

Ž 1 y ey␪ .

1

␪2

Ý xi q

2

.

Furthermore, if X denotes the truncated Poisson random variable, then EŽ X . s

xey␪␪ x

⬁

Ý Ž 1 y ey␪ . x!

xs1

s

␪ 1 y ey␪

.

Thus E H L* Ž x, ␪ . s E sy

s

⭸ 2 L* Ž x, ␪ . ⭸␪ 2 1

n␪

␪ 2 1 y ey␪

q

ney␪ Ž 1 q ␪ . y n

␪ Ž 1 y ey␪ .

2

ney␪

Ž 1 y ey␪ .

2

.

Suppose now we have the sample 1, 2, 3, 4, 5, 6 from this distribution. Let ␪ˆ0 s 1.5118 be an initial estimate of ␪ . Several iterations are made by applying formula Ž8.77., and the results are shown in Table 8.6. The final

375

MAXIMUM LIKELIHOOD ESTIMATION AND THE EM ALGORITHM

Table 8.6. Fisher’s Method of Scoring for the Truncated Poisson Distribution Iteration 1 2 3 4

⭸ L*

⭸ 2 L*

⭸␪

⭸␪ 2

y685.5137 y62.0889 y0.2822 0.0012

y1176.7632 y1696.2834 y1750.5906 y1750.8389

␪

L*

1.5118 0.9293 0.8927 0.8925

y1545.5549 y1303.3340 y1302.1790 y1302.1792

Source: Everitt Ž1987, page 31.. Reproduced with permission of Chapman and Hall, London.

estimate of ␪ is 0.8925, which is considered to be the maximum likelihood estimate of ␪ for the given sample. The convergence criterion used here is < ␪ˆiq1 y ␪ˆi < - 0.001. 8.8.1. The EM Algorithm The EM algorithm is a general iterative procedure for maximum likelihood estimation in incomplete data problems. This encompasses situations involving missing data, or when the actual data are viewed as forming a subset of a larger system of quantities. The term EM was introduced by Dempster, Laird, and Rubin Ž1977.. The reason for this terminology is that each iteration in this algorithm consists of two steps called the expectation step Ž E-step . and the maximization step Ž M-step.. In the E-step, the conditional expectations of the missing data are found given the observed data and the current estimates of the parameters. These expected values are then substituted for the missing data and used to complete the data. In the M-step, maximum likelihood estimation of the parameters is performed in the usual manner using the completed data. More generally, missing sufficient statistics can be estimated rather than the individual missing data. The estimated parameters are then used to reestimate the missing data Žor missing sufficient statistics ., which in turn lead to new parameter estimates. This defines an iterative procedure, which can be carried out until convergence is achieved. More details concerning the theory of the EM algorithm can be found in Dempster, Laird, and Rubin Ž1977., and in Little and Rubin Ž1987, Chapter 7.. The following two examples, given in the latter reference, illustrate the application of this algorithm: EXAMPLE 8.8.2. ŽLittle and Rubin, 1987, pages 130᎐131.. Consider a sample of size n from a normal distribution with a mean ␮ and a variance ␴ 2 . Suppose that x 1 , x 2 , . . . , x m are observed data and that X mq1 , X mq2 , . . . , X n are missing data. Let x obs s Ž x 1 , x 2 , . . . , x m .⬘. For i s m q 1, m q 2, . . . , n, the expected value of X i given x obs and ␪ s Ž ␮ , ␴ 2 .⬘ is ␮. Now, from Example 7.11.3, the log-likelihood function for the complete data

376

OPTIMIZATION IN STATISTICS

set is L* Ž x, ␪ . s y

n 2

log Ž 2␲␴ 2 . y

1 2␴ 2

žÝ n

is1

n

/

x i2 y 2 ␮ Ý x i q n ␮2 , Ž 8.78 . is1

where x s Ž x 1 , x 2 , . . . , x n .⬘. We note that Ý nis1 X i2 and Ý nis1 X i are sufficient statistics. Therefore, to apply the E-step of the algorithm, we only have to find the conditional expectations of these statistics given x obs and the current estimate of ␪. We thus have

žÝ n

E

is1

E

ž

/ /

Xi < ˆ ␪ j , x obs s

n

Ý X i2 < ˆ␪ j , x obs

is1

m

Ý x i q Ž n y m . ␮ˆ j ,

j s 0, 1, 2, . . . , Ž 8.79 .

is1 m

s

Ý x i2 q Ž n y m . ž ␮ˆ 2j q ␴ˆj 2 / ,

j s 0, 1, 2, . . . , Ž 8.80 .

is1

where ˆ ␪j s Ž ␮ ˆ j , ␴ˆj 2 .⬘ is the estimate of ␪ at the jth iteration with ˆ␪ 0 being an initial estimate. From Section 7.11 we recall that the maximum likelihood estimates of ␮ and ␴ 2 based on the complete data set are Ž1rn.Ý nis1 x i and Ž1rn.Ý nis1 x i2 y wŽ1rn.Ý nis1 x i x2 . Thus in the M-step, these same expressions are used, except that the current expectations of the sufficient statistics in formulas Ž8.79. and Ž8.80. are substituted for the missing data portion of the sufficient statistics. In other words, the estimates of ␮ and ␴ 2 at the Ž j q 1.th iteration are given by

␮ ˆ jq1 s 2 ␴ˆjq1 s

1 n 1 n

m

Ý x i q Ž n y m . ␮ˆ j

j s 0, 1,2, . . . ,

,

Ž 8.81 .

is1 m

Ý x i2 q Ž n y m . ž ␮ˆ 2j q ␴ˆj 2 /

y␮ ˆ 2jq1 ,

j s 0, 1, 2, . . . . Ž 8.82 .

is1

By setting ␮ ˆjs␮ ˆ jq1 s ␮ ˆ and ␴ˆj s ␴ˆjq1 s ␴ˆ in equations Ž8.81. and Ž8.82., we find that the iterations converge to

␮ ˆs ␴ˆ 2 s

1 m 1 m

m

Ý xi ,

is1 m

Ý x i2 y ␮ˆ 2 ,

is1

which are the maximum likelihood estimates of ␮ and ␴ 2 from x obs .

377

MAXIMUM LIKELIHOOD ESTIMATION AND THE EM ALGORITHM

The EM algorithm is unnecessary in this example, since the maximum likelihood estimates of ␮ and ␴ 2 can be obtained explicitly. EXAMPLE 8.8.3. ŽLittle and Rubin, 1987, pages 131᎐132.. This example was originally given in Dempster, Laird, and Rubin Ž1977.. It involves a multinomial x s Ž x 1 , x 2 , x 3 , x 4 .⬘ with cell probabilities Ž 12 y 12 ␪ , 14 ␪ , 14 ␪ , 12 .⬘, where 0 F ␪ F 1. Suppose that the observed data consist of x obs s Ž38, 34, 125.⬘ such that x 1 s 38, x 2 s 34, x 3 q x 4 s 125. The likelihood function for the complete data is L Ž x, ␪ . s

Ž x1 q x 2 q x 3 q x 4 . ! x1 ! x 2 ! x 3 ! x 4 !

Ž 12 y 12 ␪ . Ž 14 ␪ . Ž 14 ␪ . Ž 12 . x1

x2

x3

x4

.

The log-likelihood function is of the form L* Ž x, ␪ . s log

Ž x1 q x 2 q x 3 q x 4 . ! x1 ! x 2 ! x 3 ! x 4 !

q x 1 log Ž 12 y 12 ␪ .

q x 2 log Ž 14 ␪ . q x 3 log Ž 14 ␪ . q x 4 log Ž 12 . . By differentiating L*Žx, ␪ . with respect to ␪ and equating the derivative to zero we obtain y

x1 1y␪

q

x2

␪

q

x3

␪

s 0.

Hence, the maximum likelihood estimate of ␪ for the complete data set is

␪ˆs

x2 q x3 x1 q x 2 q x 3

Ž 8.83 .

.

Let us now find the conditional expectations of X 1 , X 2 , X 3 , X 4 given the observed data and the current estimate of ␪ : E Ž X 1 < ␪ˆi , x obs . s 38, E Ž X 2 < ␪ˆi , x obs . s 34, E Ž X 3 < ␪ˆi , x obs . s E Ž X 4 < ␪ˆi , x obs . s

125 Ž 14 ␪ˆi . 1 2

q 14 ␪ˆi

125 Ž 12 . 1 2

q 14 ␪ˆi

.

,

378

OPTIMIZATION IN STATISTICS

Table 8.7. The EM Algorithm for Example 8.8.3 Iteration

␪ˆ

0 1 2 3 4 5 6 7 8

0.500000000 0.608247423 0.624321051 0.626488879 0.626777323 0.626815632 0.626820719 0.626821395 0.626821484

Source: Little and Rubin Ž1987, page 132.. Reproduced with permission of John Wiley & Sons, Inc.

Thus at the Ž i q 1.st iteration we have

␪ˆiq1 s

34 q Ž 125 . Ž 14 ␪ˆi . r Ž 12 q 14 ␪ˆi .

38 q 34 q Ž 125 . Ž 14 ␪ˆi . r Ž 12 q 14 ␪ˆi .

,

Ž 8.84 .

as can be seen from applying formula Ž8.83. using the conditional expectation of X 3 instead of x 3 . Formula Ž8.84. can be used iteratively to obtain the maximum likelihood estimate of ␪ on the basis of the observed data. Using an initial estimate ␪ˆ0 s 12 , the results of this iterative procedure are given in Table 8.7. Note that if we set ␪ˆiq1 s ␪ˆi s ␪ˆ in formula Ž8.84. we obtain the quadratic equation, 197␪ˆ2 y 15␪ˆy 68 s 0 whose only positive root is ␪ˆs 0.626821498, which is very close to the value obtained in the last iteration in Table 8.7.

8.9. MINIMUM NORM QUADRATIC UNBAISED ESTIMATION OF VARIANCE COMPONENTS Consider the linear model c

ysX␣q

Ý Ui ␤ i ,

Ž 8.85 .

is1

where y is a vector of n observations; ␣ is a vector of fixed effects; ␤ 1 , ␤ 2 , . . . , ␤ c are vectors of random effects; X, U1 , U2 , . . . , Uc are known matrices of constants with ␤ c s ⑀ , the vector of random errors; and Uc s I n . We assume that the ␤ i ’s are uncorrelated with zero mean vectors and variance᎐covariance matrices ␴i 2 I m i , where m i is the number of columns of

MINIMUM NORM QUADRATIC UNBIASED ESTIMATION OF VARIANCE

379

Ui Ž i s 1, 2, . . . , c .. The variances ␴ 12 , ␴ 22 , . . . , ␴c2 are referred to as variance components. Model Ž8.85. can be written as y s X ␣ q U␤ ,

Ž 8.86 .

where U s wU1: U2 : ⭈⭈⭈ : Uc x, ␤ s Ž␤X1 , ␤X2 , . . . , ␤Xc .⬘. From model Ž8.86. we have E Ž y. s X ␣ , Var Ž y . s

c

Ý ␴i 2 Vi ,

Ž 8.87 .

is1

with Vi s UiUiX . Let us consider the estimation of a linear function of the variance components, namely, Ýcis1 a i ␴i 2 , where the a i ’s are known constants, by a quadratic estimator of the form y⬘Ay. Here, A is a symmetric matrix to be determined so that y⬘Ay satisfies certain criteria, which are the following: 1. Translation In®ariance. If instead of ␣ we consider ␥ s ␣ y ␣ 0 , then from model Ž8.86. we have y y X ␣ 0 s X ␥ q U␤. In this case, Ýcis1 a i ␴i 2 is estimated by Žy y X ␣ 0 .⬘AŽy y X ␣ 0 .. The estimator y⬘Ay is said to be translation invariant if y⬘Ays Ž y y X ␣ 0 . ⬘A Ž y y X ␣ 0 . . In order for this to be true we must have AX s 0.

Ž 8.88 .

2. Unbiasedness. E Žy⬘Ay. s Ýcis1 a i ␴i 2 . Using a result in Searle Ž1971, Theorem 1, page 55., the expected value of the quadratic form y⬘Ay is given by E Ž y⬘Ay. s ␣ ⬘X⬘AX ␣ q tr A Var Ž y . ,

Ž 8.89 .

since E Žy. s X ␣ . From formulas Ž8.87., Ž8.88., and Ž8.89. we then have E Ž y⬘Ay. s

c

Ý ␴i 2 tr Ž AVi . .

Ž 8.90 .

is1

By comparison with Ýcis1 a i ␴i 2 , the condition for unbiasedness is a i s tr Ž AVi . ,

i s 1, 2, . . . , c.

Ž 8.91 .

380

OPTIMIZATION IN STATISTICS

3. Minimum Norm. If ␤ 1 , ␤ 2 , . . . , ␤ c in model Ž8.85. were observable, then a natural unbaised estimator of Ýcis1 a i ␴i 2 would be Ýcis1 a i ␤Xi ␤ irm i , since E Ž␤Xi ␤ i . s trŽI m i ␴i 2 . s m i ␴i 2 , i s 1, 2, . . . , c. This estimator can be written as ␤⬘⌬␤, where ⌬ is the block-diagonal matrix ⌬ s Diag

ž

a1 m1

I m1 ,

a2

I m2 , . . . ,

m2

ac mc

/

I mc .

The difference between this estimator and y⬘Ay is y⬘Ayy ␤⬘⌬␤ s ␤⬘ Ž U⬘AU y ⌬ . ␤ , since AX s 0. This difference can be made small by minimizing the Euclidean norm 5 U⬘AU y ⌬ 5 2 . The quadratic estimator y⬘Ay is said to be a minimum norm quadratic unbiased estimator ŽMINQUE. of Ýcis1 a i ␴i 2 if the matrix A is determined so that 5 U⬘AU y ⌬ 5 2 attains a minimum subject to the conditions given in formulas Ž8.88. and Ž8.91.. Such an estimator was introduced by Rao Ž1971, 1972.. The minimization of 5 U⬘AU y ⌬ 5 2 is equivalent to that of trŽAVAV., where Vs Ýcis1Vi . The reason for this is the following: 5 U⬘AU y ⌬ 5 22 s tr Ž U⬘AU y ⌬ . Ž U⬘AU y ⌬ . s tr Ž U⬘AUU⬘AU . y 2 tr Ž U⬘AU⌬ . q tr Ž ⌬2 . . Ž 8.92 . Now, tr Ž U⬘AU⌬ . s tr Ž AU⌬U⬘ .

ž žÝ žÝ

c

ai

is1

mi

s tr A Ý Ui s tr s tr s

s

c

ai

is1

mi

c

ai

is1

mi

c

ai

is1

mi

c

a2i

is1

mi

Ý Ý

I m iUiX

AUiUiX AVi

/

/

/

tr Ž AVi .

,

s tr Ž ⌬2 . .

by Ž 8.91 .

MINIMUM NORM QUADRATIC UNBIASED ESTIMATION OF VARIANCE

381

Formula Ž8.92. can then be written as 5 U⬘AU y ⌬ 5 22 s tr Ž U⬘AUU⬘AU . y tr Ž ⌬2 . s tr Ž AVAV. y tr Ž ⌬2 . , since Vs Ýcis1Vi s Ýcis1UiUiX s UU⬘. The trace of ⌬2 does not involve A; hence the problem of MINQUE reduces to finding A that minimizes trŽAVAV. subject to conditions Ž8.88. and Ž8.91.. Rao Ž1971. showed that the solution to this optimization problem is of the form c

As

Ý ␭ i RVi R,

Ž 8.93 .

is1

where y

R s Vy1 y Vy1 X Ž X⬘Vy1 X . X⬘Vy1 with ŽX⬘Vy1 X.y being a generalized inverse of X⬘Vy1 X, and the ␭ i ’s are obtained from solving the equations c

Ý ␭ i tr Ž RVi RVj . s a j ,

j s 1, 2, . . . , c,

is1

which can be expressed as ␭⬘Ss a⬘,

Ž 8.94 .

where ␭ s Ž ␭1 , ␭2 , . . . , ␭ c .⬘, S is the c = c matrix Ž si j . with si j s trŽRVi RVj ., and a s Ž a1 , a2 , . . . , a c .⬘. The MINQUE of Ýcis1 a i ␴i 2 can then be written as y⬘

ž

c

/

Ý ␭ i RVi R y s

is1

c

Ý ␭ i y⬘RVi Ry

is1

s ␭⬘q, where q s Ž q1 , q2 , . . . , qc .⬘ with qi s y⬘RVi Ry Ž i s 1, 2, . . . , c .. But, from formula Ž8.94., ␭⬘ s a⬘Sy, where Sy is a generalized inverse of S. Hence, ␭⬘q s a⬘Sy q s a⬘␴, ˆ where ␴ ˆ s Ž ␴ˆ12 , ␴ˆ22 , . . . , ␴ˆc2 .⬘ is a solution of the equation S␴ ˆ s q.

Ž 8.95 .

This equation has a unique solution if and only if the individual variance components are unbiasedly estimable Žsee Rao, 1972, page

382

OPTIMIZATION IN STATISTICS

114.. Thus the MINQUEs of the ␴i 2 ’s are obtained from solving equation Ž8.95.. If the random effects in model Ž8.85. are assumed to be normally distributed, then the MINQUEs of the variance components reduce to the so-called minimum variance quadratic unbiased estimators ŽMIVQUEs.. An example that shows how to compute these estimators in the case of a random one-way classification model is given in Swallow and Searle Ž1978.. See also Milliken and Johnson Ž1984, Chapter 19..

´ CONFIDENCE INTERVALS 8.10. SCHEFFE’S Consider the linear model

Ž 8.96 .

y s X␤ q ⑀ ,

where y is a vector of n observations, X is a known matrix of order n = p and rand r ŽF p ., ␤ is a vector of unknown parameters, and ⑀ is a random error vector. It is assumed that ⑀ has the normal distribution with a mean 0 and a variance᎐covariance matrix ␴ 2 I n . Let ␺ s a⬘␤ be an estimable linear function of the elements of ␤. By this we mean that there exists a linear function t⬘y of y such that E Žt⬘y. s ␺ , where t is some constant vector. A necessary and sufficient condition for ␺ to be estimable is that a⬘ belongs to the row space of X, that is, a⬘ is a linear combination of the rows of X Žsee, for example, Searle, 1971, page 181.. Since the rank of X is r, the row space of X, denoted by ␳ ŽX., is an r-dimensional subspace of the p-dimensional Euclidean space R p. Thus a⬘␤ estimable if and only if a⬘ g ␳ ŽX.. Suppose that a⬘ is an arbitrary vector in a q-dimensional subspace L of ˆ s a⬘ŽX⬘X.y X⬘y is the best linear unbiased ␳ ŽX., where q F r. Then a⬘␤ estimator of a⬘␤, and its variance is given by y

ˆ . s ␴ 2 a⬘ Ž X⬘X . a, Var Ž a⬘␤ where ŽX⬘X.y is a generalized inverse of X⬘X Žsee, for example, Searle, 1971, ˆ and a⬘ŽX⬘X.y a are invariant to the choice of pages 181᎐182.. Both a⬘␤ y ŽX⬘X. , since a⬘␤ is estimable Žsee, for example, Searle, 1971, page 181.. In particular, if r s p, then X⬘X is of full rank and ŽX⬘X.ys ŽX⬘X.y1 . Theorem 8.10.1. Simultaneous Ž1 y ␣ .100% confidence intervals on a⬘␤ for all a⬘ g L , where L is a q-dimensional subspace of ␳ ŽX., are of the form

ˆ . Ž q MSE F␣ , q , nyr . a⬘␤

1r2

y

a⬘ Ž X⬘X . a

1r2

,

Ž 8.97 .

SCHEFFE ´’S CONFIDENCE INTERVALS

383

where F␣ , q, nyr is the upper ␣ 100th percentile of the F-distribution with q and n y r degrees of freedom, and MSE is the error mean square given by MSE s

1 nyr

y

y⬘ I n y X Ž X⬘X . X⬘ y.

Ž 8.98 .

In Theorem 8.10.1, the word ‘‘simultaneous’’ means that with probability 1 y ␣ , the values of a⬘␤ for all a⬘ g L satisfy the double inequality

ˆ y Ž q MSE F␣ , q , nyr . a⬘␤

1r2

y

a⬘ Ž X⬘X . a

ˆ q Ž q MSE F␣ , q , nyr . F a⬘␤ F a⬘␤

1r2

1r2

y

a⬘ Ž X⬘X . a

1r2

.

Ž 8.99 .

A proof of this theorem is given in Scheffe ´ Ž1959, Section 3.5.. Another proof is presented here using the method of Lagrange multipliers. This proof is based on the following lemma: Lemma 8.10.1. Let C be the set
x g R q < x⬘Ax F 14 , where A is a positive definite matrix of order q = q. Then x g C if and only if < l⬘x < F Žl⬘Ay1 l.1r2 for all l g R q. Proof. Suppose that x g C. Since A is positive definite, the boundary of C is an ellipsoid in a q-dimensional space. For any l g R q, let e be a unit vector in its direction. The projection of x on an axis in the direction of l is given by e⬘x. Consider optimizing e⬘x with respect to x over the set C. The minimum and maximum values of e⬘x are obviously determined by the end points of the projection of C on the l-axis. This is equivalent to optimizing e⬘x subject to the constraint x⬘Ax s 1, since the projection of C on the l-axis is the same as the projection of its boundary, the ellipsoid x⬘Ax s 1. This constrained optimization problem can be solved by using the method of Lagrange multipliers. Let G s e⬘x q ␭Žx⬘Ax y 1., where ␭ is a Lagrange multiplier. By differentiating G with respect to x 1 , x 2 , . . . , x q , where x i is the ith element of x Ž i s 1, 2, . . . , q ., and equating the derivatives to zero, we obtain the equation e q 2 ␭ Ax s 0, whose solution is x s yŽ1r2 ␭.Ay1 e. If we substitute this value of x into the equation x⬘Ax s 1 and then solve for ␭, we obtain the two solutions ␭1 s y 12 Že⬘Ay1 e.1r2 , ␭2 s 12 Že⬘Ay1 e.1r2 . But, e⬘x s y2 ␭, since x⬘Ax s 1. It follows that the minimum and maximum values of e⬘x under the constraint x⬘Ax s 1 are yŽe⬘Ay1 e.1r2 and Že⬘Ay1 e.1r2 , respectively. Hence, < e⬘x < F Ž e⬘Ay1 e .

1r2

.

Ž 8.100 .

Since l s 5l 5 2 e, where 5l 5 2 is the Euclidean norm of l, multiplying the two

384

OPTIMIZATION IN STATISTICS

sides of inequality Ž8.100. by 5l 5 2 yields < l⬘x < F Ž l⬘Ay1 l .

1r2

Ž 8.101 .

.

Vice versa, if inequality Ž8.101. is true for all l g R q, then by choosing l⬘ s x⬘A we obtain < x⬘Ax < F Ž x⬘AAy1Ax .

1r2

,

which is equivalent to x⬘Ax F 1, that is, x g C.

I

Proof of Theorem 8.10.1. Let L be a q = p matrix of rank q whose rows form a basis for the q-dimensional subspace L of ␳ ŽX.. Since y in model ˆ s LŽX⬘X.y X⬘y is distributed as Ž8.96. is distributed as N ŽX␤, ␴ 2 I n ., L␤ 2 y N wL␤, ␴ LŽX⬘X. L⬘x. Thus the random variable y

Fs

ˆ y ␤ . ⬘ L Ž X⬘X . L⬘ LŽ␤

y1

ˆ y␤. LŽ␤

q MSE

has the F-distribution with q and n y r degrees of freedom Žsee, for example, Searle, 1971, page 190.. It follows that P Ž F F F␣ , q , nyr . s 1 y ␣ .

Ž 8.102 .

ˆ y ␤ . and A s By applying Lemma 8.10.1 to formula Ž8.102. with x s LŽ␤ wLŽX⬘X.y L⬘xy1 rŽ q MSE F␣ , q, nyr . we obtain the equivalent probability statement

½

ˆ y ␤ . F Ž q MSE F␣ , q , nyr . P l⬘L Ž ␤

1r2

y

l⬘L Ž X⬘X . L⬘l

1r2

5

᭙l g R q s 1 y ␣ .

Let a⬘ s l⬘L. We then have

½

ˆ y ␤ . F Ž q MSE F␣ , q , nyr . P a⬘ Ž ␤

1r2

y

a⬘ Ž X⬘X . a

1r2

5

᭙a⬘ g L s 1 y ␣ .

We conclude that the values of a⬘␤ satisfy the double inequality Ž8.99. for all a⬘ g L with probability 1 y ␣ . Simultaneous Ž1 y ␣ .100% confidence intervals on a⬘␤ are therefore given by formula Ž8.97.. We refer to these intervals as Scheffe’s ´ confidence intervals. Theorem 8.10.1 can be used to obtain simultaneous confidence intervals on all contrasts among the elements of ␤. By definition, the linear function p a⬘␤ is a contrast among the elements of ␤ if Ý is1 a i s 0, where a i is the ith Ž . element of a i s 1, 2, . . . , p . If a⬘ is in the row space of X, then it must belong to a q-dimensional subspace of ␳ ŽX., where q s r y 1. Hence, simul-

SCHEFFE ´’S CONFIDENCE INTERVALS

385

taneous Ž1 y ␣ .100% confidence intervals on all such contrasts can be obtained from formula Ž8.97. by replacing q with r y 1. I 8.10.1. The Relation of Scheffe’s ´ Confidence Intervals to the F-Test There is a relationship between the confidence intervals Ž8.97. and the F-test used to test the hypothesis H0 : L␤ s 0 versus Ha : L␤ / 0, where L is the matrix whose rows form a basis for the q-dimensional subspace L of ␳ ŽX.. The test statistic for testing H0 is given by Žsee Searle, 1971, Section 5.5. y

Fs

ˆ ␤⬘L⬘ L Ž X⬘X . L⬘

y1

ˆ L␤

q MSE

,

which under H0 has the F-distribution with q and n y r degrees of freedom. The hypothesis H0 can be rejected at the ␣-level of significance if F ) F␣ , q, nyr . In this case, by Lemma 8.10.1, there exits at least one l g R q such that

ˆ < ) Ž q MSE F␣ , q , nyr . < l⬘L␤

1r2

y

l⬘L Ž X⬘X . L⬘l

1r2

.

Ž 8.103 .

It follows that the F-test rejects H0 if and only if there exists a linear ˆ where a⬘ s l⬘L for some l g R q, for which the confidence combination a⬘␤, ˆ is interval in formula Ž8.97. does not contain the value zero. In this case, a⬘␤ said to be significantly different from zero. It is easy to see that inequality Ž8.103. holds for some l g R q if and only if sup lgR q

ˆ< 2 < l⬘L␤ y

l⬘L Ž X⬘X . L⬘l

) q MSE F␣ , q , nyr ,

or equivalently, sup lgR q

l⬘G 1 l l⬘G 2 l

Ž 8.104 .

) q MSE F␣ , q , nyr ,

where

Ž 8.105 .

ˆˆ G 1 s L␤␤⬘L⬘, y

G 2 s L Ž X⬘X . L⬘.

Ž 8.106 .

However, by Theorem 2.3.17, sup lgR q

l⬘G 1 l l⬘G 2 l

s emax Ž Gy1 2 G1 . y

ˆ s ␤⬘L⬘ L Ž X⬘X . L⬘

y1

ˆ, L␤

Ž 8.107 .

386

OPTIMIZATION IN STATISTICS

y1 . where emax ŽGy1 2 G 1 is the largest eigenvalue of G 2 G 1 . The second equality ˆˆ in Ž8.107. is true because the nonzero eigenvalues of wLŽX⬘X.y L⬘xy1 L␤␤⬘L⬘ ˆ wLŽX⬘X.y L⬘xy1 L␤ ˆ by Theorem 2.3.9. Note that are the same as those of ␤⬘L⬘ the latter expression is the numerator sum of squares of the F-test statistic for H0 . Ž y1 . The eigenvector of Gy1 2 G 1 corresponding to e max G 2 G 1 is of special interest. Let l* be such an eigenvector. Then

l*⬘G 1 l* l*⬘G 2 l*

s emax Ž Gy1 2 G1 . .

Ž 8.108 .

This follows from the fact that l* satisfies the equation

Ž G 1 y emax G 2 . l* s 0, . where emax is an abbreviation for emax ŽGy1 2 G 1 . It is easy to see that l* can be y1 ˆ chosen to be the vector G 2 L␤, since y1 ˆ y1 ˆ ˆ y1 ˆ Gy1 2 G 1 Ž G 2 L␤ . s G 2 L␤␤⬘L⬘ Ž G 2 L␤ .

ž

/

y1 ˆ y1 ˆ ˆ s ␤⬘L⬘G 2 L␤ G 2 L␤

ˆ s emax Gy1 2 L␤. y1 ˆ This shows that Gy1 2 L␤ is an eigenvector of G 2 G 1 for the eigenvalue e max . Ž . Ž . From inequality 8.104 and formula 8.108 we conclude that if the F-test rejects H0 at the ␣-level, then

ˆ < ) Ž q MSE F␣ , q , nyr . < l*⬘L␤

1r2

y

l*⬘L Ž X⬘X . L⬘l*

1r2

.

ˆ where a*⬘ s l*⬘L, is signifiThis means that the linear combination a*⬘␤, ˆ as cantly different from zero. Let us express a*⬘␤ q

ˆs l*⬘L␤

Ý lUi ␥ˆi ,

Ž 8.109 .

is1

ˆ respectively where lUi and ␥ ˆi are the ith elements of l* and ␥ˆ s L␤, Ž i s 1, 2, . . . , q .. If we divide ␥ by its estimated standard error ␬ˆi wwhich is ˆi equal to the square root of the ith diagonal element of the variance᎐covariˆ namely, ␴ 2 LŽX⬘X.y L⬘ with ␴ 2 replaced by the error ance matrix of L␤, mean square MSE in formula Ž8.98.x, then formula Ž8.109. can be written as q

ˆs l*⬘L␤

Ý lUi ␬ˆiˆ␶ i ,

Ž 8.110 .

is1

␶ i s ␥ˆir␬ˆi , i s 1, 2, . . . , q. Consequently, large values of < lUi < ␬ˆi identify where ˆ those elements of ␥ ˆ that are influential contributors to the significance of

SCHEFFE ´’S CONFIDENCE INTERVALS

387

the F-test concerning H0 . Note that the elements of ␥ s L␤ form a set of linearly independent estimable linear functions of ␤. We conclude from the previous arguments that the eigenvector l*, which corresponds to the largest eigenvalue of Gy1 2 G 1 , can be conveniently used to identify an estimable linear function of ␤ that is significantly different from zero whenever the F-test rejects H0 . It should be noted that if model Ž8.96. is a response surface model Žin this case, the matrix X in the model is of full column rank, that is, r s p . whose input variables, x 1 , x 2 , . . . , x k , have different units of measurement, then these variables must be made scale free. This is accomplished as follows: If x ui denotes the uth measurement on x i , then we may consider the transformation

z ui s

x ui y x i si

,

i s 1, 2, . . . , k ; u s 1, 2, . . . , n,

where x i s Ž1rn.Ý nus1 x ui , si s wÝ nus1 Ž x ui y x i . 2 x1r2 , and n is the total number of observations. One advantage of this scaling convention, besides making the input variables scale free, is that it can greatly improve the conditioning of the matrix X with regard to multicollinearity Žsee, for example, Belsley, Kuh, and Welsch, 1980, pages 183᎐185.. EXAMPLE 8.10.1.

Let us consider the one-way classification model

yi j s ␮ q ␣ i q ⑀ i j ,

i s 1, 2, . . . , m; j s 1, 2, . . . , n i ,

Ž 8.111 .

where ␮ and ␣ i are unknown parameters with the latter representing the effect of the ith level of a certain factor at m levels; n i observations are obtained at the ith level. The ⑀ i j ’s are random errors assumed to be independent and normally distributed with zero means and a common variance ␴ 2 . Model Ž8.111. can be represented in vector form as model Ž8.96.. Here, y s Ž y 11 , y 12 , . . . , y 1 n 1, y 21 , y 22 , . . . , y 2 n 2 , . . . , ym1 , ym 2 , . . . , ym n m .⬘, ␤ s Ž ␮ , ␣ 1 , ␣ 2 , . . . , ␣ m .⬘, and X is of order n = Ž m q 1. of the form X s w1 n : Tx, where 1 n is a vector of ones of order n = 1, n s Ý m is1 n i , and T s DiagŽ1 n1, 1 n 2 , . . . , 1 n m .. The rank of X is r s m. For such a model, the hypothesis of interest is H0 : ␣ 1 s ␣ 2 s ⭈⭈⭈ s ␣ m , which can be expressed as H0 : L␤ s 0, where L is a matrix of order

388

OPTIMIZATION IN STATISTICS

Ž m y 1. = Ž m q 1. and rank m y 1 of the form 0 0 L s .. . 0

1 1 . . . 1

y1 0 . . . 0

0 y1 . . . 0

⭈⭈⭈ ⭈⭈⭈ ⭈⭈⭈

0 0 . . . . y1

This hypothesis states that the factor under consideration has no effect on the response. Note that each row of L is a linear combination of the rows of X. For example, the ith row of Lᎏwhose elements are equal to zero except for the second and the Ž i q 2.th elements, which are equal to 1 and y1, respectivelyᎏis the difference between rows 1 and ␯ i q 1 of X, where ␯ i s Ýijs1 n j , i s 1,2, . . . , m y 1. Thus the rows of L form a basis for a q-dimensional subspace L of ␳ ŽX., the row space of X, where q s m y 1. Let ␮ i s ␮ q ␣ i . Then ␮ i is the mean of the ith level of the factor m Ž i s 1, 2, . . . , m.. Consider the contrast ␺ s Ý m is1 c i ␮ i , that is, Ý is1 c i s 0. We Ž . can write ␺ s a⬘␤, where a⬘ s 0, c1 , c 2 , . . . , c m belongs to a q-dimensional subspace of R mq1. This subspace is the same as L , since each row of L is of the form Ž0, c1 , c 2 , . . . , c m . with Ý m is1 c i s 0. Vice versa, if ␺ s a⬘␤ is such that a⬘ s Ž0, c1 , c 2 , . . . , c m . with Ý m is1 c i s 0, then a⬘ can be expressed as a⬘ s Ž yc2 ,y c 3 , . . . , yc m . L, since c1 s yÝ m is2 c i . Hence, a⬘ g L . It follows that L is a subspace associated with all contrasts among the means ␮ 1 , ␮ 2 , . . . , ␮ m of the m levels of the factor. Simultaneous Ž1 y ␣ .100% confidence intervals on all contrasts of the Ž . form ␺ s Ý m is1 c i ␮ i can be obtained by applying formula 8.97 . Here, q s m y 1, r s m, and a generalized inverse of X⬘X is of the form y 0 Ž X⬘X . s

0

0⬘ , D

y1 y1 . where D s DiagŽ ny1 and 0 is a zero vector of order m = 1. 1 , n2 , . . . , nm Hence, y

ˆ s Ž 0, c1 , c2 , . . . , c m . Ž X⬘X . X⬘y a⬘␤ m

s

Ý c i yi . ,

is1

SCHEFFE ´’S CONFIDENCE INTERVALS

389

i where yi.s Ž1rn i .Ý njs1 yi j , i s 1, 2, . . . , m. Furthermore,

y

a⬘ Ž X⬘X . a s

m

c i2

is1

ni

Ý

.

By making the substitution in formula Ž8.97. we obtain m

Ý c i yi .. Ž m y 1. MSE F␣ , my1 , nym

is1

1r2

žÝ / m

c i2

is1

ni

1r2

.

Ž 8.112 .

Now, if the F-test rejects H0 at the ␣-level, then there exists a contrast U ˆ Ým is1 c i yi.s a*⬘␤, which is significantly different from zero, that is, the interval Ž8.112. for c i s cUi Ž i s 1, 2, . . . , m. does not contain the value zero. Here, y1 ˆ a*⬘ s l*⬘L, where l* s Gy1 2 L␤ is an eigenvector of G 2 G 1 corresponding to y1 emam ŽG 2 G 1 .. We have that y

L Ž X⬘X . L⬘

Gy1 2 s

y1

s

ž

1 n1

J my1 q ⌳

/

y1

,

where J my1 is a matrix of ones of order Ž m y 1. = Ž m y 1., and ⌳ s y1 y1 . DiagŽ ny1 2 , n 3 , . . . , n m . By applying the Sherman᎐Morrison᎐Woodbury formula Žsee Exercise 2.15., we obtain X Gy1 2 s n1 Ž n1 ⌳ q 1 my1 1 my1 .

s ⌳y1 y

y1

⌳y1 1 my1 1Xmy1 ⌳y1 n1 q 1Xmy1 ⌳y1 1 my1

n2 1 n3 . w n2 , n3 , . . . , nm x . s Diag Ž n 2 , n 2 , . . . , n m . y n .. nm Also, y 1.y y 2. y

ˆ s L Ž X⬘X . X⬘y s ␥ ˆ s L␤

y 1.y y 3. . . . . y 1.y ym .

ˆ It can be verified that the ith element of l* s Gy1 2 L␤ is given by lUi s n iq1 Ž y 1.y yiq1 . . y

n iq1 n

m

Ý n j Ž y1.y yj. . ,

js2

i s 1, 2, . . . , m y 1. Ž 8.113 .

390

OPTIMIZATION IN STATISTICS

The estimated standard error, ␬ ˆi , of the ith element of ␥ˆ Ž i s 1, 2, . . . , m y 1. is the square root of the ith diagonal element of MSE LŽX⬘X.y L⬘ s wŽ1rn1 .J my1 q ⌳ x MSE , that is,

ž

␬ˆi s

1

q

n1

/

1 n iq1

1r2

MSE

i s 1, 2, . . . , m y 1.

,

Thus by formula Ž8.110., large values of < lUi < ␬ ˆi identify those elements of ˆ that are influential contributors to the significance of the F-test. In ␥ ˆ s L␤ particular, if the data set used to analyze model Ž8.111. is balanced, that is, n i s nrm for i s 1, 2, . . . , m, then < lUi < ␬ ˆi s < yiq1 .y y . . <

ž

2n m

MSE

/

1r2

i s 1, 2, . . . , m y 1,

,

where y. .s Ž1rm.Ý m is1 yi. . ˆ can be expressed as Alternatively, the contrast aU ⬘␤ y

ˆ s l*⬘L Ž X⬘X . X⬘y a*⬘␤

ž

my1

s 0, s

Ý

lUi ,y lU1 ,y lU2 , . . . , ylUmy1

is1 m

/Ž

0, y 1. , y 2. , . . . , ym . . ⬘

Ý cUi yi . ,

is1

where lUi is given in formula Ž8.113. and

° l, Ý s~ ¢yl , my1

cUi

U j

i s 1,

Ž 8.114 .

js1

U iy1

i s 2, 3, . . . , m.

Since the estimated standard error of yi. is Ž MSE rn i .1r2 , i s 1, 2, . . . , m, by dividing yi. by this value we obtain m

ˆs a*⬘␤

Ý

is1

ž

1 ni

MSE

/

1r2

cUi wi ,

where wi s yi. rŽ MSE rn i .1r2 is a scaled value of yi. Ž i s 1, 2, . . . , m.. Hence, large values of Ž MSE rn i .1r2 < cUi < identify those yi.’s that contribute significantly to the rejection of H0 . In particular, for a balanced data set, lUi s

n m

Ž y . .y yiq1 . . ,

i s 1, 2, . . . , m y 1.

391

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

Thus from formula Ž8.114. we get

ž / MSE ni

1r2

< cUi < s

ž

n m

MSE

/

1r2

< yi .y y . . < ,

i s 1, 2, . . . , m.

We conclude that large values of < yi.y y. . < are responsible for the rejection of H0 by the F-test. This is consistent with the fact that the numerator sum of Ž . 2 when squares of the F-test statistic for H0 is proportional to Ý m is1 yi.y y. . the data set is balanced.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Adby, P. R., and M. A. H. Dempster Ž1974.. Introduction of Optimization Methods. Chapman and Hall, London. ŽThis book is an introduction to nonlinear methods of optimization. It covers basic optimization techniques such as steepest descent and the Newton᎐Raphson method.. Ash, A., and A. Hedayat Ž1978.. ‘‘An introduction to design optimality with an overview of the literature.’’ Comm. Statist. Theory Methods, 7, 1295᎐1325. Atkinson, A. C. Ž1982.. ‘‘Developments in the design of experiments.’’ Internat. Statist. Re®., 50, 161᎐177. Atkinson, A. C. Ž1988.. ‘‘Recent developments in the methods of optimum and related experimental designs.’’ Internat. Statist. Re®., 56, 99᎐115. Bates, D. M., and D. G. Watts Ž1988.. Nonlinear Regression Analysis and its Applications. Wiley, New York. ŽEstimation of parameters in a nonlinear model is addressed in Chaps. 2 and 3. Design aspects for nonlinear models are briefly discussed in Section 3.14.. Bayne, C. K., and I. B. Rubin Ž1986.. Practical Experimental Designs and Optimization Methods for Chemists. VCH Publishers, Deerfield Beach, Florida. ŽSteepest ascent and the simplex method are discussed in Chap. 5. A bibliography of optimization and response surface methods, as actually applied in 17 major fields of chemistry, is provided in Chap. 7.. Belsley, D. A., E. Kuh, and R. E. Welsch Ž1980.. Regression Diagnostics. Wiley, New York. ŽChap. 3 is devoted to the diagnosis of multicollinearity among the columns of the matrix in a regression model. Multicollinearity renders the model’s least-squares parameter estimates less precise and less useful than would otherwise be the case.. Biles, W. E., and J. J. Swain Ž1980.. Optimization and Industrial Experimentation. Wiley-Interscience, New York. ŽChaps. 4 and 5 discuss optimization techniques that are directly applicable in response surface methodology.. Bohachevsky, I. O., M. E. Johnson, and M. L. Stein Ž1986.. ‘‘Generalized simulated annealing for function optimization.’’ Technometrics, 28, 209᎐217. Box, G. E. P. Ž1982.. ‘‘Choice of response surface design and alphabetic optimality.’’ Utilitas Math., 21B, 11᎐55. Box, G. E. P., and D. W. Behnken Ž1960.. ‘‘Some new three level designs for the study of quantitative variables.’’ Technometrics, 2, 455᎐475.

392

OPTIMIZATION IN STATISTICS

Box, G. E. P., and N. R. Draper Ž1959.. ‘‘A basis for the selection of a response surface design.’’ J. Amer. Statist. Assoc., 55, 622᎐654. Box, G. E. P., and N. R. Draper Ž1963.. ‘‘The choice of a second order rotatable design.’’ Biometrika, 50, 335᎐352. Box, G. E. P., and N. R. Draper Ž1965.. ‘‘The Bayesian estimation of common parameters from several responses.’’ Biometrika, 52, 355᎐365. Box, G. E. P., and N. R. Draper Ž1987.. Empirical Model-Building and Response Surfaces. Wiley, New York. ŽChap. 9 introduces the exploration of maxima with second-order models; the alphabetic optimality approach is critically considered in Chap. 14. Many examples are given throughout the book.. Box, G. E. P., and H. L. Lucas Ž1959.. ‘‘Design of experiments in nonlinear situations.’’ Biometrika, 46, 77᎐90. Box, G. E. P., and K. B. Wilson Ž1951.. ‘‘On the experimental attainment of optimum conditions.’’ J. Roy. Statist. Soc. Ser. B, 13, 1᎐45. Bunday, B. D. Ž1984.. Basic Optimization Methods. Edward Arnold Ltd., Victoria, Australia. ŽChaps. 3 and 4 discuss basic optimization techniques such as the Nelder᎐Mead simplex method and the Davidon᎐Fletcher ᎐Powell method.. Conlon, M. Ž1991.. ‘‘The controlled random search procedure for function optimization.’’ Personal communication. ŽThis is a FORTRAN file for implementing Price’s controlled random search procedure. . Conlon, M., and A. I. Khuri Ž1992.. ‘‘Multiple response optimization.’’ Technical Report, Department of Statistics, University of Florida, Gainesville, Florida. Cook, R. D., and C. J. Nachtsheim Ž1980.. ‘‘A comparison of algorithms for constructing exact D-optimal designs.’’ Technometrics, 22, 315᎐324. Dempster, A. P., N. M. Laird, and D. B. Rubin Ž1977.. ‘‘Maximum likelihood from incomplete data via the EM algorithm.’’ J. Roy. Statist. Soc. Ser. B, 39, 1᎐38. Draper, N. R. Ž1963.. ‘‘Ridge analysis of response surfaces.’’ Technometrics, 5, 469᎐479. Everitt, B. S. Ž1987.. Introduction to Optimization Methods and Their Application in Statistics. Chapman and Hall, London. ŽThis book gives a brief introduction to optimization methods and their use in several areas of statistics. These include maximum likelihood estimation, nonlinear regression estimation, and applied multivariate analysis. . Fedorov, V. V. Ž1972.. Theory of Optimal Experiments. Academic Press, New York. ŽThis book is a translation of a monograph in Russian. It presents the mathematical apparatus of experimental design for a regression model.. Fichtali, J., F. R. Van De Voort, and A. I. Khuri Ž1990.. ‘‘Multiresponse optimization of acid casein production.’’ J. Food Process Eng., 12, 247᎐258. Fletcher, R. Ž1987.. Practical Methods of Optimization, 2nd ed. Wiley, New York. ŽThis book gives a detailed study of several unconstrained and constrained optimization techniques. . Fletcher, R., and M. J. D. Powell Ž1963.. ‘‘A rapidly convergent descent method for minimization.’’ Comput. J., 6, 163᎐168. Hartley, H. O., and J. N. K. Rao Ž1967.. ‘‘Maximum likelihood estimation for the mixed analysis of variance model.’’ Biometrika, 54, 93᎐108.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

393

Hoerl, A. E. Ž1959.. ‘‘Optimum solution of many variables equations.’’ Chem. Eng. Prog., 55, 69᎐78. Huber, P. J. Ž1973.. ‘‘Robust regression: Asymptotics, conjectures and Monte Carlo.’’ Ann. Statist., 1, 799᎐821. Huber, P. J. Ž1981.. Robust Statistics. Wiley, New York. ŽThis book gives a solid foundation in robustness in statistics. Chap. 3 introduces and discusses M-estimation; Chap. 7 addresses M-estimation for a regression model.. Johnson, M. E., and C. J. Nachtsheim Ž1983.. ‘‘Some guidelines for constructing exact D-optimal designs on convex design spaces.’’ Technometrics, 25, 271᎐277. Jones, E. R., and T. J. Mitchell Ž1978.. ‘‘Design criteria for detecting model inadequacy.’’ Biometrika, 65, 541᎐551. Karson, M. J., A. R. Manson, and R. J. Hader Ž1969.. ‘‘Minimum bias estimation and experimental design for response surfaces.’’ Technometrics, 11, 461᎐475. Khuri, A. I., and M. Conlon Ž1981.. ‘‘Simultaneous optimization of multiple responses represented by polynomial regression functions.’’ Technometric, 23, 363᎐375. Khuri, A. I., and J. A. Cornell Ž1996.. Response Surfaces, 2nd ed. Marcel Dekker, New York. ŽOptimization techniques in response surface methodology are discussed in Chap. 5.. Khuri, A. I., and R. H. Myers Ž1979.. ‘‘Modified ridge analysis.’’ Technometrics, 21, 467᎐473. Khuri, A. I., and H. Sahai Ž1985.. ‘‘Variance components analysis: A selective literature survey.’’ Internat. Statist. Re®., 53, 279᎐300. Kiefer, J. Ž1958.. ‘‘On the nonrandomized optimality and the randomized nonoptimality of symmetrical designs.’’ Ann. Math. Statist., 29, 675᎐699. Kiefer, J. Ž1959.. ‘‘Optimum experimental designs’’ Žwith discussion .. J. Roy. Statist. Soc. Ser. B, 21, 272᎐319. Kiefer, J. Ž1960.. ‘‘Optimum experimental designs V, with applications to systematic and rotatable designs.’’ In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1. University of California Press, Berkeley, pp. 381᎐405. Kiefer, J. Ž1961.. ‘‘Optimum designs in regression problems II.’’ Ann. Math. Statist., 32, 298᎐325. Kiefer, J. Ž1962a.. ‘‘Two more criteria equivalent to D-optimality of designs.’’ Ann. Math. Statist., 33, 792᎐796. Kiefer, J. Ž1962b.. ‘‘An extremum result.’’ Canad. J. Math., 14, 597᎐601. Kiefer, J. Ž1975.. ‘‘Optimal design: Variation in structure and performance under change of criterion.’’ Biometrika, 62, 277᎐288. Kiefer, J., and J. Wolfowitz Ž1960.. ‘‘The equivalence of two extremum problems.’’ Canad. J. Math., 12, 363᎐366. Kirkpatrick, S., C. D. Gelatt, and M. P. Vechhi Ž1983.. ‘‘Optimization by simulated annealing.’’ Science, 220, 671᎐680. Little, R. J. A., and D. B. Rubin Ž1987.. Statistical Analysis with Missing Data. Wiley, New York. ŽThe theory of the EM algorithm is introduced in Chap. 7. The book presents a systematic approach to the analysis of data with missing values, where inferences are based on likelihoods derived from formal statistical models for the data. .

394

OPTIMIZATION IN STATISTICS

Lucas, J. M. Ž1976.. ‘‘Which response surface design is best.’’ Technometrics, 18, 411᎐417. Miller, R. G., Jr. Ž1981.. Simultaneous Statistical Inference, 2nd ed. Springer-Verlag, New York. ŽScheffe’s ´ simultaneous confidence intervals are derived in Chap. 2.. Milliken, G. A., and D. E. Johnson Ž1984.. Analysis of Messy Data. Lifetime Learning Publications, Belmont, California. ŽThis book presents several techniques and methods for analyzing unbalanced data. . Mitchell, T. J. Ž1974.. ‘‘An algorithm for the construction of D-optimal experimental designs.’’ Technometrics, 16, 203᎐210. Myers, R. H. Ž1976.. Response Surface Methodology. Author, Blacksburg, Virginia. ŽChap. 5 discusses the determination of optimum operating conditions in response surface methodology; designs for fitting first-order and second-order models are discussed in Chaps. 6 and 7, respectively; Chap. 9 presents the J-criterion for choosing a response surface design.. Myers, R. H. Ž1990.. Classical and Modern Regression with Applications, 2nd ed., PWS-Kent, Boston. ŽChap. 3 discusses the effects and hazards of multicollinearity in a regression model. Methods for detecting and combating multicollinearity are given in Chap. 8.. Myers, R. H., and W. H. Carter, Jr. Ž1973.. ‘‘Response surface techniques for dual response systems.’’ Technometrics, 15, 301᎐317. Myers, R. H., and A. I. Khuri Ž1979.. ‘‘A new procedure for steepest ascent.’’ Comm. Statist. Theory Methods, 8, 1359᎐1376. Myers, R. H., A. I. Khuri, and W. H. Carter, Jr. Ž1989.. ‘‘Response surface methodology: 1966᎐1988.’’ Technometrics, 31, 137᎐157. Nelder, J. A., and R. Mead Ž1965.. ‘‘A simplex method for function minimization.’’ Comput. J., 7, 308᎐313. Nelson, L. S. Ž1973.. ‘‘A sequential simplex procedure for non-linear least-squares estimation and other function minimization problems.’’ In 27th Annual Technical Conference Transaction, American Society for Quality Control, pp. 107᎐117. Olsson, D. M., and L. S. Nelson Ž1975.. ‘‘The Nelder᎐Mead simplex procedure for function minimization.’’ Technometrics, 17, 45᎐51. Pazman, A. Ž1986.. Foundations of Optimum Experimental Design. D. Reidel, Dordrecht, Holland. Plackett, R. L., and J. P. Burman Ž1946.. ‘‘The design of optimum multifactorial experiments.’’ Biometrika, 33, 305᎐325. Price, W. L. Ž1977.. ‘‘A controlled random search procedure for global optimization.’’ Comput. J., 20, 367᎐370. Rao, C. R. Ž1970.. ‘‘Estimation of heteroscedastic variances in linear models.’’ J. Amer. Statist. Assoc., 65, 161᎐172. Rao, C. R. Ž1971.. ‘‘Estimation of variance and covariance componentsᎏMINQUE theory.’’ J. Multi®ariate Anal., 1, 257᎐275. Rao, C. R. Ž1972.. ‘‘Estimation of variance and covariance components in linear models.’’ J. Amer. Statist. Assoc., 67, 112᎐115. Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts.

395

EXERCISES

Rustagi, J. S., ed. Ž1979.. Optimizing Methods in Statistics. Academic Press, New York. Scheffe, ´ H. Ž1959.. The Analysis of Variance. Wiley, New York. ŽThis classic book presents the basic theory of analysis of variance, mainly in the balanced case.. Searle, S. R. Ž1971.. Linear Models. Wiley, New York. ŽThis book describes general procedures of estimation and hypothesis testing for linear models. Estimable linear functions for models that are not of full rank are discussed in Chap. 5.. Seber, G. A. F. Ž1984.. Multi®ariate Obser®ations, Wiley, New York. ŽThis book gives a comprehensive survey of the subject of multivariate analysis and provides many useful references. . Silvey, S. D. Ž1980.. Optimal Designs. Chapman and Hall, London. Spendley, W., G. R. Hext, and F. R. Himsworth Ž1962.. ‘‘Sequential application of simplex designs in optimization and evolutionary operation.’’ Technometrics, 4, 441᎐461. St. John, R. C., and N. R. Draper Ž1975.. ‘‘ D-Optimality for regression designs: A review.’’ Technometrics, 17, 15᎐23. Swallow, W. H., and S. R. Searle Ž1978.. ‘‘Minimum variance quadratic unbiased estimation ŽMIVQUE. of variance components.’’ Technometrics, 20, 265᎐272. Watson, G. S. Ž1964.. ‘‘A note on maximum likelihood.’’ Sankhya Ser. A, 26, 303᎐304. Wynn, H. P. Ž1970.. ‘‘The sequential generation of D-optimum experimental designs.’’ Ann. Math. Statist., 41, 1655᎐1664. Wynn, H. P. Ž1972.. ‘‘Results in the theory and construction of D-optimum experimental designs.’’ J. Roy. Statist. Soc. Ser. B, 34, 133᎐147. Zanakis, S. H., and J. S. Rustagi, eds. Ž1982.. Optimization in Statistics. North-Holland, Amsterdam, Holland. ŽThis is Volume 19 in Studies in the Management Sciences. It contains 21 articles that address applications of optimization in three areas of statistics, namely, regression and correlation; multivariate data analysis and design of experiments; and statistical estimation, reliability, and quality control..

EXERCISES 8.1. Consider the function f Ž x 1 , x 2 . s 8 x 12 y 4 x 1 x 2 q 5 x 22 . Minimize f Ž x 1 , x 2 . using the method of steepest descent with x 0 s Ž5, 2.⬘ as an initial point. 8.2. Conduct a simulated steepest ascent exercise as follows: Use the function

␩ Ž x 1 , x 2 . s 47.9 q 3 x 1 y x 2 q 4 x 12 q 4 x 1 x 2 q 3 x 22 as the true mean response, which depends on two input variables x 1 and x 2 . Generate response values by using the model y Ž x. s ␩ Ž x. q ⑀ ,

396

OPTIMIZATION IN STATISTICS

where ⑀ has the normal distribution with mean 0 and variance 2.25, and x s Ž x 1 , x 2 .⬘. Fit a first-order model in x 1 and x 2 in a neighborhood of the origin using a 2 2 factorial design along with the corresponding simulated response values. Make sure that replications are taken at the origin in order to test for lack of fit of the fitted model. Determine the path of steepest ascent, then proceed along it using simulated response values. Conduct additional experiments as described in Section 8.3.1. 8.3. Two types of fertilizers were applied to experimental plots to assess their effects on the yield of a certain variety of potato. The design settings used in the experiment along with the corresponding yield values are given in the following table:

Fertilizer 1

Fertilizer 2

x1

x2

Yield y Žlbrplot.

50.0 120.0 50.0 120.0 35.5 134.5 85.0 85.0 85.0

15.0 15.0 25.0 25.0 20.0 20.0 12.9 27.1 20.0

y1 1 y1 1 y21r2 21r2 0 0 0

y1 y1 1 1 0 0 y21r2 21r2 0

24.30 35.82 40.50 50.94 30.60 42.90 22.50 50.40 45.69

Original Settings

Coded Settings

(a) Fit a second-order model in the coded variables x1 s

F1 y 85 35

,

x2 s

F2 y 20 5

to the yield data, where F1 and F2 are the original settings of fertilizers 1 and 2, respectively, used in the experiment. (b) Apply the method of ridge analysis to determine the settings of the two fertilizers that are needed to maximize the predicted yield Žin the space of the coded input variables, the region R is the interior and boundary of a circle centered at the origin with a radius equal to 2 1r2 .. 8.4. Suppose that ␭1 and ␭ 2 are two values of the Lagrange multiplier ␭ used in the method of ridge analysis. Let ˆ y 1 and ˆ y 2 be the corresponding values of ˆ y on the two spheres x⬘x s r 12 and x⬘x s r 22 , respectively. Show that if r 1 s r 2 and ␭1 ) ␭2 , then ˆ y1 )ˆ y2 .

397

EXERCISES

8.5. Consider again Exercise 8.4. Let x 1 and x 2 be the stationary points corresponding to ␭1 and ␭2 , respectively. Consider also the matrix

ˆ y ␭i I. , MŽ x i . s 2 ŽB

i s 1, 2.

Show that if r 1 s r 2 , MŽx 1 . is positive definite, and MŽx 2 . is indefinite, then ˆ y1 -ˆ y2 . 8.6. Consider once more the method of ridge analysis. Let x be a stationary point that corresponds to the radius r. (a) Show that

r

3

⭸ 2r ⭸␭2

k

s2r

2

Ý

is1

ž / ⭸ xi

2

⭸␭

k

q r

2

Ý

is1

ž / žÝ ⭸ xi ⭸␭

2

k

y

is1

xi

⭸ xi ⭸␭

/

2

,

where x i is the ith element of x Ž i s 1, 2, . . . , k .. (b) Make use of part Ža. to show that

⭸ 2r ⭸␭2

)0

if r / 0.

8.7. Suppose that the ‘‘true’’ mean response ␩ Žx. is represented by a model of order d 2 in k input variables x 1 , x 2 , . . . , x k of the form

␩ Ž x . s f⬘ Ž x . ␤ q g⬘ Ž x . ␦ , where x s Ž x 1 , x 2 , . . . , x k .⬘. The fitted model is of order d1 Ž- d 2 . of the form

ˆy Ž x . s f⬘ Ž x . ˆ␭ , where ˆ␭ is an estimator of ␤, not necessarily obtained by the method of least squares. Let ␥ s E Žˆ␭ .. (a) Give an expression for B, the average squared bias of ˆ y Žx., in terms of ␤, ␦, and ␥. (b) Show that B achieves its minimum value if and only if ␥ is of the y1 form ␥ s C␶, where ␶ s Ž␤⬘, ␦⬘.⬘ and C s wI: ⌫11 ⌫12 x. The matrices ⌫11 and ⌫12 are the region moments used in formula Ž8.54.. (c) Deduce from part Žb. that B achieves its minimum value if and only if C␶ is an estimable linear function Žsee Searle, 1971, Section 5.4..

398

OPTIMIZATION IN STATISTICS

(d) Use part Žc. to show that B achieves its minimum value if and only if there exists a matrix L such that C s LwX: Zx, where X and Z are matrices consisting of the values taken by f⬘Žx. and g⬘Žx., respectively, at n experimental runs. (e) Deduce from part Žd. that B achieves its minimum for any design for which the row space of wX: Zx contains the rows of C. (f) Show that if ˆ␭ is the least-squares estimator of ␤, that is, ˆ␭ s ŽX⬘X.y1 X⬘y, where y is the vector of response values at the n experimental runs, then the design property stated in part Že. holds for any design that satisfies the conditions described in equations Ž8.56.. w Note: This problem is based on an article by Karson, Manson, and Hader Ž1969., who introduced the so-called minimum bias estimation to minimize the average squared bias B.x 8.8. Consider again Exercise 8.7. Suppose that f⬘Žx.␤ s ␤ 0 q Ý3is1 ␤i x i is a first-order model in three input variables fitted to a data set obtained by using the design yg g Ds g yg

yg g yg g

yg yg , g g

where g is a scale factor. The region of interest is a sphere of radius 1. Suppose that the ‘‘true’’ model is of the form

␩ Ž x. s ␤0 q

3

Ý ␤i x i q ␤12 x 1 x 2 q ␤13 x 1 x 3 q ␤ 23 x 2 x 3 .

is1

(a) Can g be chosen so that D satisfies the conditions described in equations Ž8.56.? (b) Can g be chosen so that D satisfies the minimum bias property described in part Že. of Exercise 8.7? 8.9. Consider the function h Ž ␦ , D . s ␦⬘⌬ ␦ , where ␦ is a vector of unknown parameters as in model Ž8.48., ⌬ is the X matrix in formula Ž8.53., namely ⌬ s A⬘⌫11 A y ⌫12 A y A⬘⌫12 q ⌫22 , and D is the design matrix.

399

EXERCISES

(a) Show that for a given D, the maximum of hŽ ␦, D. over the region ␺ s
␦ < ␦⬘␦ F r 2 4 is equal to r 2 emax Ž ⌬ ., where emax Ž ⌬ . is the largest eigenvalue of ⌬. (b) Deduce from part Ža. a design criterion for choosing D. 8.10. Consider fitting the model y Ž x . s f⬘ Ž x . ␤ q ⑀ , where ⑀ is a random error with a zero mean and a variance ␴ 2 . Suppose that the ‘‘true’’ mean response is given by

␩ Ž x . s f⬘ Ž x . ␤ q g⬘ Ž x . ␦. Let X and Z be the same matrices defined in part Žd. of Exercise 8.7. Consider the function ␭Ž ␦, D. s ␦⬘S␦, where Ss Z⬘ I y X Ž X⬘X .

y1

X⬘ Z,

and D is the design matrix. The quantity ␭Ž ␦, D.r␴ 2 is the noncentrality parameter associated with the lack of fit F-test for the fitted model Žsee Khuri and Cornell, 1996, Section 2.6.. Large values of ␭r␴ 2 increase the power of the lack of fit test. By formula Ž8.54., the minimum value of B is given by Bmin s

n

␴2

␦⬘T ␦ ,

X y1 where T s ⌫22 y ⌫12 ⌫11 ⌫12 . The fitted model is considered to be inadequate if there exists some constant ␬ ) 0 such that ␦⬘T ␦ G ␬ . Show that

inf ␦⬘S␦ s ␬ emin Ž Ty1 S. ,

␦g⌽

where emin ŽTy1 S. is the smallest eigenvalue of Ty1 S and ⌽ is the region
␦ < ␦⬘T ␦ G ␬ 4 . w Note: On the basis of this problem, we can define a new design criterion, that which maximizes emin ŽTy1 S. with respect to D. A design chosen according to this criterion is called ⌳ 1-optimal Žsee Jones and Mitchell, 1978..x 8.11. A second-order model of the form y Ž x . s ␤ 0 q ␤ 1 x 1 q ␤ 2 x 2 q ␤ 11 x 12 q ␤ 22 x 22 q ␤ 12 x 1 x 2 q ⑀

400

OPTIMIZATION IN STATISTICS

is fitted using a rotatable central composite design D, which consists of a factorial 2 2 portion, an axial portion with an axial parameter ␣ s 2 1r2 , and n 0 center-point replications. The settings of the 2 2 factorial portion are "1. The region of interest R consists of the interior and boundary of a circle of radius 2 1r2 centered at the origin. (a) Express V, the average variance of the predicted response given by formula Ž8.52., as a function of n 0 . (b) Can n 0 be chosen so that it minimizes V ? 8.12. Suppose that we have r response functions represented by the models yi s X␤ i q ⑀ i ,

i s 1, 2, . . . , r ,

where X is a known matrix of order n = p and rank p. The random error vectors have the same variance᎐covariance structure as in Section 8.7. Let Fs E ŽY. s XB, where Y s wy1:y2 : ⭈⭈⭈ :yr x and B s w ␤ 1: ␤ 2 : ⭈⭈⭈ :␤ r x. Show that the determinant of ŽY y F.⬘ŽY y F. attains a minimum ˆ where B ˆ is obtained by replacing each ␤ i in B with value when B s B, ˆ i s ŽX⬘X.y1 X⬘yi Ž i s 1, 2, . . . , r .. ␤ w Note: The minimization of the determinant of ŽY y F.⬘ŽY y F. with respect to B represents a general multiresponse estimation criterion known as the Box᎐Draper determinant criterion Žsee Box and Draper, 1965..x 8.13. Let A be a p= p matrix with nonnegative eigenvalues. Show that det Ž A . F exp tr Ž A y I p . . w Note: This inequality is proved in an article by Watson Ž1964.. It is based on the simple inequality aF expŽ ay 1., which can be easily proved for any real number a.x 8.14. Let x 1 , x 2 , . . . , x n be a sample of n independently distributed random vectors from a p-variate normal distribution N Ž␮, V.. The corresponding likelihood function is Ls

1

Ž 2␲ .

n pr2

det Ž V.

nr2

exp y

1 2

n

Ý Ž x i y ␮ . ⬘Vy1 Ž x i y ␮ .

.

is1

It is known that the maximum likelihood estimate of ␮ is x, where x s Ž1rn.Ý nis1 x i Žsee, for example, Seber, 1984, pages 59᎐61.. Let S be the matrix Ss

1 n

n

Ý Ž x i y x . Ž x i y x . ⬘.

is1

401

EXERCISES

Show that S is the maximum likelihood estimate of V by proving that 1 det Ž V. F

nr2

exp y 1

det Ž S.

nr2

1 2

n

Ý Ž x i y x . ⬘Vy1 Ž x i y x .

is1

exp y

1 2

n

Ý Ž x i y x . ⬘Sy1 Ž x i y x .

,

is1

or equivalently, det Ž SVy1 .

nr2

exp y

n 2

tr Ž SVy1 . F exp y

n 2

tr Ž I p . .

w Hint: Use the inequality given in Exercise 8.13.x 8.15. Consider the random one-way classification model yi j s ␮ q ␣ i q ⑀ i j ,

i s 1, 2, . . . , a; j s 1, 2, . . . , n i ,

where the ␣ i ’s and ⑀ i j ’s are independently distributed as N Ž0, ␴␣2 . and N Ž0, ␴⑀ 2 .. Determine the matrix S and the vector q in equation Ž8.95. that can be used to obtain the MINQUEs of ␴␣2 and ␴⑀ 2 . 8.16. Consider the linear model y s X␤ q ⑀ , where X is a known matrix of order n = p and rank p, and ⑀ is normally distributed with a zero mean vector and a variance᎐covariy Žx. denote the predicted response at a point x ance matrix ␴ 2 I n . Let ˆ in a region of interest R. Use Scheffe’s ´ confidence intervals given by formula Ž8.97. to obtain simultaneous confidence intervals on the mean response values at the points x 1 , x 2 , . . . , x m Ž m F p . in R. What is the joint confidence coefficient for these intervals? 8.17. Consider the fixed-effects two-way classification model yi jk s ␮ q ␣ i q ␤ j q Ž ␣␤ . i j q ⑀ i jk , i s 1, 2, . . . , a; j s 1, 2, . . . , b; k s 1, 2, . . . , m,

402

OPTIMIZATION IN STATISTICS

where ␣ i and ␤ j are unknown parameters, Ž ␣␤ . i j is the interaction effect, and ⑀ i jk is a random error that has the normal distribution with a zero mean and a variance ␴ 2 . (a) Use Scheffe’s ´ confidence intervals to obtain simultaneous confidence intervals on all contrasts among the ␮ i ’s, where ␮ i s E Ž yi. . . and yi. .s Ž1rbm.Ý bjs1 Ý m ks1 y i jk . (b) Identify those yi. .’s that are influential contributors to the significance of the F-test concerning the hypothesis H0 : ␮ 1 s ␮ 2 s ⭈⭈⭈ s ␮ a .

CHAPTER 9

Approximation of Functions

The class of polynomials is undoubtedly the simplest class of functions. In this chapter we shall discuss how to use polynomials to approximate continuous functions. Piecewise polynomial functions Žsplines . will also be discussed. Attention will be primarily confined to real-valued functions of a single variable x.

9.1. WEIERSTRASS APPROXIMATION We may recall from Section 4.3 that if a function f Ž x . has derivatives of all orders in some neighborhood of the origin, then it can be represented by a power series of the form Ý⬁ns0 a n x n. If ␳ is the radius of convergence of this series, then the series converges uniformly for < x < F r, where r - ␳ Žsee Theorem 5.4.4.. It follows that for a given ⑀ ) 0 we can take sufficiently many terms of this power series and obtain a polynomial pnŽ x . s Ý nks0 a k x k of degree n for which < f Ž x . y pnŽ x .< - ⑀ for < x < F r. But a function that is not differentiable of all orders does not have a power series representation. However, if the function is continuous on the closed interval w a, b x, then it can be approximated uniformly by a polynomial. This is guaranteed by the following theorem: Theorem 9.1.1 ŽWeierstrass Approximation Theorem.. Let f : w a, b x ™ R be a continuous function. Then, for any ⑀ ) 0, there exists a polynomial pŽ x . such that f Ž x . y pŽ x . - ⑀

for all x g w a, b x .

Proof. Without loss of generality we can consider w a, b x to be the interval w0, 1x. This can always be achieved by making a change of variable of the form ts

xy a by a

. 403

404

APPROXIMATION OF FUNCTIONS

As x varies from a to b, t varies from 0 to 1. Thus, if necessary, we consider that such a linear transformation has been made and that t has been renamed as x. For each n, let bnŽ x . be defined as a polynomial of degree n of the form n

Ý

bn Ž x . s

ks0

ž /

ž/

k n k nyk x Ž1yx. f , k n

Ž 9.1 .

where

ž/

n! n s . k k! Ž n y k . !

We have that n

Ž 9.2 .

k n k nyk x Ž1yx. s x, n k

Ž 9.3 .

ks0 n

Ý

ks0

ž/ ž/ ž/

n k nyk x Ž1yx. s 1, k

Ý

ž

/

k2 n k 1 2 x nyk x Ž1yx. s 1y x q . 2 k n n n

n

Ý

ks0

Ž 9.4 .

These identities can be shown as follows: Let Yn be a binomial random variable B Ž n, x .. Thus Yn represents the number of successes in a sequence of n independent Bernoulli trials with x the probability of success on a single trial. Hence, E Ž Yn . s nx and VarŽ Yn . s nx Ž1 y x . Žsee, for example, Harris, 1966, page 104; see also Exercise 5.30.. It follows that n

Ý

ks0

ž/

n k nyk x Ž1yx. s P Ž 0 F Yn F n . s 1. k

Ž 9.5 .

Furthermore, n

Ýk

ks0 n

Ý k2

ks0

ž/

n k nyk x Ž1yx. s E Ž Yn . s nx, k

ž/

n k nyk x Ž1yx. s E Ž Yn2 . s Var Ž Yn . q E Ž Yn . k s nx Ž 1 y x . q n2 x 2 s n2

Identities Ž9.2. ᎐ Ž9.4. follow directly from Ž9.5. ᎐ Ž9.7..

ž

1y

Ž 9.6 . 2

1 n

/

x2q

x n

. Ž 9.7 .

405

WEIERSTRASS APPROXIMATION

Let us now consider the difference f Ž x . y bnŽ x ., which with the help of identity Ž9.2. can be written as n

f Ž x . y bn Ž x . s

Ý

f Ž x. yf

ks0

ž /ž / k

n k nyk x Ž1yx. . k

n

Ž 9.8 .

Since f Ž x . is continuous on w0, 1x, then it must be bounded and uniformly continuous there Žsee Theorems 3.4.5 and 3.4.6.. Hence, for the given ⑀ ) 0, there exist numbers ␦ and m such that f Ž x1 . y f Ž x 2 . F

⑀

if x 1 y x 2 - ␦

2

and for all xg w 0, 1 x .

f Ž x. -m From formula Ž9.8. we then have f Ž x . y bn Ž x . F

n

Ý

f Ž x. yf

ks0

ž /ž / k

n k nyk x Ž1yx. . k

n

If < xy krn < - ␦ , then < f Ž x . y f Ž krn.< - ⑀r2; otherwise, we have < f Ž x . y f Ž krn.< - 2 m for 0 F xF 1. Consequently, by using identities Ž9.2. ᎐ Ž9.4. we obtain f Ž x . y bn Ž x . F

Ý

< xykrn < -␦

q

Ý

F

2 F

⑀

⑀

< xykrn < G␦

q

q

2 F

⑀ 2

Ý

q2m

2 F

n k nyk x Ž1yx. k

2m

< xykrn < G␦

⑀

ž/ ž/

⑀ n k nyk x Ž1yx. 2 k

q

2m

␦2 2m

␦

2

< xykrn < G␦ n

Ý

ks0

2m

␦

Ý

2

ž

ž

1y

k2 n

2

1 n

Ž krny x .

2

Ž krny x .

2

ž

y

/

k n

2 kx n

x2q

qx2 x

n

n k nyk x Ž1yx. k

/ž / 2

yx

ž/

n k nyk x Ž1yx. k

/ž /

n k nyk x Ž1yx. k

y2 x2qx2 .

406

APPROXIMATION OF FUNCTIONS

Hence, f Ž x . y bn Ž x . F F

⑀ 2 ⑀

q q

2

2 m xŽ1yx.

␦2 m 2 n␦ 2

n ,

Ž 9.9 .

since x Ž1 y x . F 14 for 0 F xF 1. By choosing n large enough that m 2 n␦

2

-

⑀ 2

,

we conclude that f Ž x . y bn Ž x . - ⑀ for all xg w0, 1x. The proof of the theorem follows by taking pŽ x . s bnŽ x ..

I

Definition 9.1.1. Let f Ž x . be defined on w0, 1x. The polynomial bnŽ x . defined by formula Ž9.1. is called the Bernstein polynomial of degree n for f Ž x .. I By the proof of Theorem 9.1.1 we conclude that the sequence
bnŽ x .4⬁ns1 of Bernstein polynomials converges uniformly to f Ž x . on w0, 1x. These polynomials are useful in that they not only prove the existence of an approximating polynomial for f Ž x ., but also provide a simple explicit representation for it. Another advantage of Bernstein polynomials is that if f Ž x . is continuously differentiable on w0, 1x, then the derivative of bnŽ x . converges also uniformly to f ⬘Ž x .. A more general statement is given by the next theorem, whose proof can be found in Davis Ž1975, Theorem 6.3.2, page 113.. Theorem 9.1.2. Let f Ž x . be p times differentiable on w0, 1x. If the pth derivative is continuous there, then lim

n™⬁

d p bn Ž x . dx p

s

d pf Ž x. dx p

uniformly on w0, 1x. Obviously, the knowledge that the sequence
bnŽ x .4⬁ns1 converges uniformly to f Ž x . on w0, 1x is not complete without knowing something about the rate of convergence. For this purpose we need to define the so-called modulus of continuity of f Ž x . on w a, b x.

407

WEIERSTRASS APPROXIMATION

Definition 9.1.2. If f Ž x . is continuous on w a, b x, then, for any ␦ ) 0, the modulus of continuity of f Ž x . on w a, b x is

␻Ž␦ . s

sup < x 1yx 2 < F␦

where x 1 and x 2 are points in w a, b x.

f Ž x1 . y f Ž x 2 . , I

On the basis of Definition 9.1.2 we have the following properties concerning the modulus of continuity: Lemma 9.1.1.

If 0 - ␦ 1 F ␦ 2 , then ␻ Ž ␦ 1 . F ␻ Ž ␦ 2 ..

Lemma 9.1.2. For a function f Ž x . to be uniformly continuous on w a, b x it is necessary and sufficient that lim ␦ ™ 0 ␻ Ž ␦ . s 0. The proofs of Lemmas 9.1.1 and 9.1.2 are left to the reader. Lemma 9.1.3.

For any ␭ ) 0, ␻ Ž ␭␦ . F Ž ␭ q 1. ␻ Ž ␦ ..

Proof. Suppose that ␭ ) 0 is given. We can find an integer n such that n F ␭ - n q 1. By Lemma 9.1.1, ␻ Ž ␭␦ . F ␻ wŽ n q 1. ␦ x. Let x 1 and x 2 be two points in w a, b x such that x 1 - x 2 and < x 1 y x 2 < F Ž n q 1. ␦ . Let us also divide the interval w x 1 , x 2 x into n q 1 equal parts, each of length Ž x 2 y x 1 .rŽ n q 1., by means of the partition points yi s x 1 q i

Ž x 2 y x1 .

i s 0, 1, . . . , n q 1.

,

nq1

Then f Ž x1 . y f Ž x 2 . s f Ž x 2 . y f Ž x1 . n

Ý

s

f Ž yiq1 . y f Ž yi .

is0 n

F

Ý f Ž yiq1 . y f Ž yi .

is0

F Ž n q 1. ␻ Ž ␦ . , since < yiq1 y yi < s w1rŽ n q 1.x< x 2 y x 1 < F ␦ for i s 0, 1, . . . , n. It follows that

␻ Ž n q 1. ␦ s

sup < x 1yx 2 < F Ž nq1 . ␦

f Ž x1 . y f Ž x 2 . F Ž n q 1. ␻ Ž ␦ . .

408

APPROXIMATION OF FUNCTIONS

Consequently,

␻ Ž ␭␦ . F ␻ Ž n q 1 . ␦ F Ž n q 1 . ␻ Ž ␦ . F Ž ␭ q 1. ␻ Ž ␦ . .

I

Theorem 9.1.3. Let f Ž x . be continuous on w0, 1x, and let bnŽ x . be the Bernstein polynomial defined by formula Ž9.1.. Then 3

f Ž x . y bn Ž x . F

␻

2

ž' / 1

n

for all xg w0, 1x, where ␻ Ž ␦ . is the modulus of continuity of f Ž x . on w0, 1x. Proof. Using formula Ž9.1. and identity Ž9.2., we have that f Ž x . y bn Ž x . s

n

Ý

ks0 n

F

Ý

f Ž x. yf

ks0 n

F

Ý␻

ks0

ž /ž / Ž ž /ž / Ž /ž / Ž . k

f Ž x. yf

ž

xy

k n

n k nyk x 1yx. k

n

k

n k nyk x 1yx. k

n

n k x 1yx k

nyk

.

Now, by applying Lemma 9.1.3 we can write

␻

ž

xy

/ ž ž

k

k

s ␻ n1r2 xy

n

n

F 1 q n1r2 xy

ny1r2 k n

/

/

␻ Ž ny1r2 . .

Thus f Ž x . y bn Ž x . F

n

Ý

ks0

ž

1 q n1r2 xy

k n

F ␻ Ž ny1r2 . 1 q n1r2

/ n

Ý

ž/ Ž ž/ Ž

␻ Ž ny1r2 .

ks0

xy

k n

n k nyk x 1yx. k

n k nyk x 1yx. . k

409

WEIERSTRASS APPROXIMATION

But, by the Cauchy᎐Schwarz inequality Žsee part 1 of Theorem 2.1.2., we have n

Ý

xy

ks0

k n

ž/

n k nyk x Ž1yx. k

n

s

Ý

xy

ks0

ž Ýž Ý

xy

ks0 n

s

n k nyk x Ž1yx. k

n

n

F

ž/

k

xy

ks0

k n k n

/ž / /ž / Ž ž / 2

2

s

F

ž/

n k nyk x Ž1yx. k

1r2

n

n k nyk x Ž1yx. k

Ý

ks0

1

n

x2q

1r2

ž/

n k nyk x Ž1yx. k

1r2

1r2

n k nyk x 1yx. k

s x2y2 x2q 1y xŽ1yx.

1r2

x n

,

by identity Ž 9.2 .

1r2

,

by identities Ž 9.3 . and Ž 9.4 .

1r2

n 1

1r2

,

4n

since x Ž 1 y x . F 14 .

It follows that f Ž x . y bn Ž x . F ␻ Ž ny1r2 . 1 q n1r2

ž / 1

4n

1r2

,

that is, f Ž x . y bn Ž x . F 32 ␻ Ž ny1r2 . for all xg w0, 1x.

I

We note that Theorem 9.1.3 can be used to prove Theorem 9.1.1 as follows: If f Ž x . is continuous on w0, 1x, then f Ž x . is uniformly continuous on w0, 1x. Hence, by Lemma 9.1.2, ␻ Ž ny1r2 . ™ 0 as n ™ ⬁. Corollary 9.1.1. w0, 1x, then

If f Ž x . is a Lipschitz continuous function LipŽ K, ␣ . on f Ž x . y bn Ž x . F 32 Kny␣ r2

for all xg w0, 1x.

Ž 9.10 .

410

APPROXIMATION OF FUNCTIONS

Proof. By Definition 3.4.6, f Ž x1 . y f Ž x 2 . F K < x1 y x 2 < ␣ for all x 1 , x 2 in w0, 1x. Thus

␻ Ž ␦ . F K ␦ ␣. By Theorem 9.1.3 we then have f Ž x . y bn Ž x . F 32 Kny␣ r2 for all xg w0, 1x.

I

Theorem 9.1.4 ŽVoronovsky’s Theorem.. If f Ž x . is bounded on w0, 1x and has a second-order derivative at a point x 0 in w0, 1x, then lim n bn Ž x 0 . y f Ž x 0 . s 12 x 0 Ž 1 y x 0 . f ⬙ Ž x 0 . .

n™⬁

Proof. See Davis Ž1975, Theorem 6.3.6, page 117..

I

We note from Corollary 9.1.1 and Voronovsky’s theorem that the convergence of Bernstein polynomials can be very slow. For example, if f Ž x . satisfies the conditions of Voronovsky’s theorem, then at every point xg w0, 1x where f ⬙ Ž x . / 0, bnŽ x . converges to f Ž x . just like crn, where c is a constant. EXAMPLE 9.1.1. We recall from Section 3.4.2 that f Ž x . s 'x is LipŽ1, 12 . for xG 0. Then, by Corollary 9.1.1,

'x y bn Ž x .

F

3 2 n1r4

,

for 0 F xF 1, where bn Ž x . s

n

Ý

ks0

ž/

ž /

k n k nyk x Ž1yx. k n

1r2

.

9.2. APPROXIMATION BY POLYNOMIAL INTERPOLATION One possible method to approximate a function f Ž x . with a polynomial pŽ x . is to select such a polynomial so that both f Ž x . and pŽ x . have the same values at a certain number of points in the domain of f Ž x .. This procedure is called interpolation. The rationale behind it is that if f Ž x . agrees with pŽ x . at some known points, then the two functions should be close to one another at intermediate points. Let us first consider the following result given by the next theorem.

411

APPROXIMATION BY POLYNOMIAL INTERPOLATION

Theorem 9.2.1. Let a0 , a1 , . . . , a n be n q 1 distinct points in R, the set of real numbers. Let b 0 , b1 , . . . , bn be any given set of n q 1 real numbers. Then, there exists a unique polynomial pŽ x . of degree F n such that pŽ a i . s bi , i s 0, 1, . . . , n. Proof. Since pŽ x . is a polynomial of degree F n, it can be represented as pŽ x . s Ý njs0 c j x j. We must then have n

Ý c j aij s bi ,

i s 0, 1, . . . , n.

js0

These equations can be written in vector form as 1 1 . . . 1

a0

a02

⭈⭈⭈

a0n

c0

a1 . . . an

a12

⭈⭈⭈

a1n

c1 b1 . s . . . . . . cn bn

. . . a2n

. . . a nn

⭈⭈⭈

b0

Ž 9.11 .

The determinant of the Ž n q 1. = Ž n q 1. matrix on the left side of equation n Ž . Ž9.11. is known as Vandermonde’s determinant and is equal to Ł i) j ai y a j . The proof of this last assertion can be found in, for example, Graybill Ž1983, Theorem 8.12.2, page 266.. Since the a i ’s are distinct, this determinant is different from zero. It follows that this matrix is nonsingular. Hence, equation Ž9.11. provides a unique solution for c 0 , c1 , . . . , c n . I Corollary 9.2.1. sented as

The polynomial pŽ x . in Theorem 9.2.1 can be repren

pŽ x . s

Ý bi l i Ž x . ,

Ž 9.12 .

is0

where n

li Ž x . s Ł js0 j/i

xy a j ai y a j

,

i s 0, 1, . . . , n.

Ž 9.13 .

Proof. We have that l i Ž x . is a polynomial of degree n Ž i s 0, 1, . . . , n.. Furthermore, l i Ž a j . s 0 if i / j, and l i Ž a i . s 1 Ž i s 0, 1, . . . , n.. It follows that Ý nis0 bi l i Ž x . is a polynomial of degree F n and assumes the values b 0 , b1 , . . . , bn at a0 , a1 , . . . , a n , respectively. This polynomial is unique by Theorem 9.2.1. I Definition 9.2.1. The polynomial defined by formula Ž9.12. is called a Lagrange interpolating polynomial. The points a0 , a1 , . . . , a n are called points

412

APPROXIMATION OF FUNCTIONS

of interpolation Žor nodes., and l i Ž x . in formula Ž9.13. is called the ith Lagrange polynomial associated with the a i ’s. I The values b 0 , b1 , . . . , bn in formula Ž9.12. are frequently the values of some function f Ž x . at the points a0 , a1 , . . . , a n . Thus f Ž x . and the polynomial pŽ x . in formula Ž9.12. attain the same values at these points. The polynomial pŽ x ., which can be written as pŽ x . s

n

Ý f Ž ai . l i Ž x . ,

Ž 9.14 .

is0

provides therefore an approximation for f Ž x . over w a0 , a n x. EXAMPLE 9.2.1. Consider the function f Ž x . s x 1r2 . Let a0 s 60, a1 s 70, a2 s 85, a3 s 105 be interpolation points. Then p Ž x . s 7.7460 l 0 Ž x . q 8.3666 l 1 Ž x . q 9.2195l 2 Ž x . q 10.2470 l 3 Ž x . , where l0 Ž x . s

Ž xy 70 . Ž x y 85. Ž x y 105. , Ž 60 y 70 . Ž 60 y 85. Ž 60 y 105.

l1Ž x . s

Ž xy 60 . Ž xy 85. Ž xy 105. , Ž 70 y 60 . Ž 70 y 85. Ž 70 y 105.

l2 Ž x . s

Ž xy 60 . Ž xy 70. Ž xy 105. , Ž 85 y 60 . Ž 85 y 70. Ž 85 y 105.

l3 Ž x . s

Ž xy 60. Ž xy 70 . Ž xy 85 . . Ž 105 y 60 . Ž 105 y 70 . Ž 105 y 85.

Table 9.1. Approximation of f ( x ) s x 1rr 2 by the Lagrange Interpolating Polynomial p( x ) x

f Ž x.

pŽ x .

60 64 68 70 74 78 82 85 90 94 98 102 105

7.74597 8.00000 8.24621 8.36660 8.60233 8.83176 9.05539 9.21954 9.48683 9.69536 9.89949 10.09950 10.24695

7.74597 7.99978 8.24611 8.36660 8.60251 8.83201 9.05555 9.21954 9.48646 9.69472 9.89875 10.09899 10.24695

413

APPROXIMATION BY POLYNOMIAL INTERPOLATION

Using pŽ x . as an approximation of f Ž x . over the interval w60, 105x, tabulated values of f Ž x . and pŽ x . were obtained at several points inside this interval. The results are given in Table 9.1. 9.2.1. The Accuracy of Lagrange Interpolation Let us now address the question of evaluating the accuracy of Lagrange interpolation. The answer to this question is given in the next theorem. Theorem 9.2.2. Suppose that f Ž x . has n continuous derivatives on the interval w a, b x, and its Ž n q 1.st derivative exists on Ž a, b .. Let as a0 - a1 ⭈⭈⭈ - a n s b be n q 1 points in w a, b x. If pŽ x . is the Lagrange interpolating polynomial defined by formula Ž9.14., then there exists a point c g Ž a, b . such that for any xg w a, b x, x/ a i Ž i s 0, 1, . . . , n., f Ž x . y pŽ x . s

1

Ž n q 1. !

f Ž nq1. Ž c . g nq1 Ž x . ,

Ž 9.15 .

where n

g nq1 Ž x . s Ł Ž xy a i . . is0

Proof. Define the function hŽ t . as hŽ t . s f Ž t . y pŽ t . y f Ž x . y pŽ x .

g nq1 Ž t . g nq1 Ž x .

.

If t s x, then hŽ x . s 0. For t s a i Ž i s 0, 1, . . . , n., h Ž ai . s f Ž ai . y p Ž ai . y f Ž x . y p Ž x .

g nq1 Ž a i . g nq1 Ž x .

s 0.

The function hŽ t . has n continuous derivatives on w a, b x, and its Ž n q 1.st derivative exists on Ž a, b .. Furthermore, hŽ t . vanishes at x and at all n q 1 interpolation points, that is, it has at least n q 2 different zeros in w a, b x. By Rolle’s theorem ŽTheorem 4.2.1., h⬘Ž t . vanishes at least once between any two zeros of hŽ t . and thus has at least n q 1 different zeros in Ž a, b .. Also by Rolle’s theorem, h⬙ Ž t . has at least n different zeros in Ž a, b .. By continuing this argument, we see that hŽ nq1. Ž t . has at least one zero in Ž a, b ., say at the point c. But, hŽ nq1. Ž t . s f Ž nq1. Ž t . y p Ž nq1. Ž t . y s f Ž nq1. Ž t . y

f Ž x . y pŽ x .

f Ž x . y pŽ x . g nq1 Ž x .

g nq1 Ž x .

Ž n q 1 . !,

Ž nq1. g nq1 Ž t.

414

APPROXIMATION OF FUNCTIONS

since pŽ t . is a polynomial of degree F n and g nq1 Ž t . is a polynomial of the form t nq1 q ␭1 t n q ␭2 t ny1 q ⭈⭈⭈ q␭ nq1 for suitable constants ␭1 , ␭ 2 , . . . , ␭ nq1 . We thus have f Ž nq1. Ž c . y

f Ž x . y pŽ x . g nq1 Ž x .

Ž n q 1 . !s 0,

from which we can conclude formula Ž9.15..

I

Corollary 9.2.2. Suppose that f Ž nq1. Ž x . is continuous on w a, b x. Let ␶nq1 s sup aF x F b < f Ž nq1. Ž x .< , ␬ nq1 s sup aF x F b < g nq1 Ž x .< . Then f Ž x . y pŽ x . F

sup

␶nq1 ␬ nq1

aFxFb

Ž n q 1. !

.

Proof. This follows directly from formula Ž9.15. and the fact that f Ž x . y pŽ x . s 0 for xs a0 , a1 , . . . , a n . I n Ž x y a i .< We note that ␬ nq1 , being the supremum of < g nq1 Ž x .< s < Ł is0 over w a, b x, is a function of the location of the a i ’s. From Corollary 9.2.2 we can then write

sup

f Ž x . y pŽ x . F

␾ Ž a0 , a1 , . . . , a n .

aFxFb

Ž n q 1. !

sup

f Ž nq1. Ž x . , Ž 9.16 .

aFxFb

where ␾ Ž a0 , a1 , . . . , a n . s sup aF x F b < g nq1 Ž x .< . This inequality provides us with an upper bound on the error of approximating f Ž x . with pŽ x . over the interpolation region. We refer to this error as interpolation error. The upper bound clearly shows that the interpolation error depends on the location of the interpolation points. Corollary 9.2.3. then

If, in Corollary 9.2.2, n s 2, and if a1 y a0 s a2 y a1 s ␦ , f Ž x . y pŽ x . F

sup aFxFb

'3 27

␶ 3␦ 3.

Proof. Consider g 3 Ž x . s Ž xy a0 .Ž x y a1 .Ž xy a2 ., which can be written as g 3 Ž x . s z Ž z 2 y ␦ 2 ., where zs xy a1. This function is symmetric with respect to xs a1. It is easy to see that < g 3 Ž x .< attains an absolute maximum over a0 F xF a2 , or equivalently, y␦ F zF ␦ , when z s "␦r '3 . Hence,

␬3 s s

sup a 0FxFa 2

␦

'3

ž

␦2 3

g 3 Ž x . s max

y␦FzF␦

y␦ 2

/

s

2 3'3

␦ 3.

zŽ z2y␦ 2 .

415

APPROXIMATION BY POLYNOMIAL INTERPOLATION

By applying Corollary 9.2.2 we obtain sup a0FxFa2

f Ž x . y pŽ x . F

'3 27

␶ 3␦ 3.

We have previously noted that the interpolation error depends on the choice of the interpolation points. This leads us to the following important question: How can the interpolation points be chosen so as to minimize the interpolation error? The answer to this question lies in inequality Ž9.16.. One reasonable criterion for the choice of interpolation points is the minimization of ␾ Ž a0 , a1 , . . . , a n . with respect to a0 , a1 , . . . , a n . It turns out that the optimal locations of a0 , a1 , . . . , a n are given by the zeros of the Chebyshev polynomial Žof the first kind. of degree n q 1 Žsee Section 10.4.1.. I Definition 9.2.2.

The Chebyshev polynomial of degree n is defined by

Tn Ž x . s cos Ž n Arccos x . sxnq

ž/

ž/

2 n ny2 2 n ny4 2 x x Ž x y 1. q Ž x y 1 . q ⭈⭈⭈ , 2 4

n s 0, 1, . . . . Ž 9.17 . Obviously, by the definition of TnŽ x ., y1 F xF 1. One of the properties of TnŽ x . is that it has simple zeros at the following n points:

␨ i s cos

Ž 2 i y 1. 2n

␲ ,

i s 1, 2, . . . , n.

I

The proof of this property is given in Davis Ž1975, pages 61᎐62.. We can consider Chebyshev polynomials defined on the interval w a, b x by making the transformation xs

aq b

q

by a

2

2

t,

which transforms the interval y1 F t F 1 into the interval aF x F b. In this case, the zeros of the Chebyshev polynomial of degree n over the interval w a, b x are given by zi s

aq b 2

q

by a 2

cos

ž

2iy1 2n

/

␲ ,

i s 1, 2, . . . , n.

We refer to the z i ’s as Chebyshev points. These points can be obtained geometrically by subdividing the semicircle over the interval w a, b x into n

416

APPROXIMATION OF FUNCTIONS

equal arcs and then projecting the midpoint of each arc onto the interval Žsee De Boor, 1978, page 26.. Chebyshev points have a very interesting property that pertains to the minimization of ␾ Ž a0 , a1 , . . . , a n . in inequality Ž9.16.. This property is described in Theorem 9.2.3, whose proof can be found in Davis Ž1975, Section 3.3.; see also De Boor Ž1978, page 30.. Theorem 9.2.3.

The function n

␾ Ž a0 , a1 , . . . , a n . s sup

⌸ Ž xy a i . ,

is0

aFxFb

where the a i ’s belong to the interval w a, b x, achieves its minimum at the zeros of the Chebyshev polynomial of degree n q 1, that is, at zi s

aq b

q

2

bya 2

cos

ž

2iq1 2nq2

/

␲ ,

i s 0, 1, . . . , n,

Ž 9.18 .

and min

a 0 , a1 , . . . , a n

␾ Ž a0 , a1 , . . . , a n . s

2 Ž by a . 4 nq1

nq1

.

From Theorem 9.2.3 and inequality Ž9.16. we conclude that the choice of the Chebyshev points given in formula Ž9.18. is optimal in the sense of reducing the interpolation error. In other words, among all sets of interpolation points of size n q 1 each, Chebyshev points produce a Lagrange polynomial approximation for f Ž x . over the interval w a, b x with a minimum upper bound on the error of approximation. Using inequality Ž9.16., we obtain the following interesting result: sup

f Ž x . y pŽ x . F

aFxFb

2

Ž n q 1. !

ž

by a 4

/

nq1

sup

f Ž nq1. Ž x . . Ž 9.19 .

aFxFb

The use of Chebyshev points in the construction of Lagrange interpolating polynomial pŽ x . for the function f Ž x . over w a, b x produces an approximation which, for all practical purposes, differs very little from the best possible approximation of f Ž x . by a polynomial of the same degree. This was shown by Powell Ž1967.. More explicitly, let p*Ž x . be the best approximating polynomial of f Ž x . of the same degree as pŽ x . over w a, b x. Then, obviously, sup aFxFb

f Ž x . y p* Ž x . F sup

f Ž x . y pŽ x . .

aFxFb

De Boor Ž1978, page 31. pointed out that for n F 20, sup aFxFb

f Ž x . y p Ž x . F 4 sup aFxFb

f Ž x . y p* Ž x . .

417

APPROXIMATION BY POLYNOMIAL INTERPOLATION

This indicates that the error of interpolation which results from the use of Lagrange polynomials in combination with Chebyshev points does not exceed the minimum approximation error by more than a factor of 4 for n F 20. This is a very useful result, since the derivation of the best approximating polynomial can be tedious and complicated, whereas a polynomial approximation obtained by Lagrange interpolation that uses Chebyshev points as interpolation points is simple and straightforward. 9.2.2. A Combination of Interpolation and Approximation In Section 9.1 we learned how to approximate a continuous function f : w a, b x ™ R with a polynomial by applying the Weierstrass theorem. In this section we have seen how to interpolate values of f on w a, b x by using Lagrange polynomials. We now show that these two processes can be combined. More specifically, suppose that we are given n q 1 distinct points in w a, b x, which we denote by a0 , a1 , . . . , a n with a0 s a and a n s b. Let ⑀ ) 0 be given. We need to find a polynomial q Ž x . such that < f Ž x . y q Ž x .< - ⑀ for all x in w a, b x, and f Ž a i . s q Ž a i ., i s 0, 1, . . . , n. By Theorem 9.1.1 there exists a polynomial pŽ x . such that f Ž x . y pŽ x . - ⑀ ⬘

for all xg w a, b x ,

where ⑀ ⬘ - ⑀rŽ1 q M ., and M is a nonnegative number to be described later. Furthermore, by Theorem 9.2.1 there exists a unique polynomial uŽ x . such that u Ž ai . s f Ž ai . y p Ž ai . ,

i s 0, 1, . . . , n.

This polynomial is given by n

uŽ x . s

Ý

f Ž ai . y p Ž ai . l i Ž x . ,

Ž 9.20 .

is0

where l i Ž x . is the ith Lagrange polynomial defined in formula Ž9.13.. Using formula Ž9.20. we obtain max u Ž x . F

aFxFb

n

Ý f Ž ai . y p Ž ai .

is0

max l i Ž x .

aFxFb

F ⑀ ⬘M, where Ms Ý nis0 max aF x F b < l i Ž x .< , which is some finite nonnegative number. Note that M depends only on w a, b x and a0 , a1 , . . . , a n . Now, define q Ž x . as q Ž x . s pŽ x . q uŽ x .. Then q Ž ai . s p Ž ai . q u Ž ai . s f Ž ai . ,

i s 0, 1, . . . , n.

418

APPROXIMATION OF FUNCTIONS

Furthermore, f Ž x . y q Ž x . F f Ž x . y p Ž x . q uŽ x . - ⑀ ⬘ q ⑀ ⬘M for all x g w a, b x .

-⑀

9.3. APPROXIMATION BY SPLINE FUNCTIONS Approximation of a continuous function f Ž x . with a single polynomial pŽ x . may not be quite adequate in situations in which f Ž x . represents a real physical relationship. The behavior of such a function in one region may be unrelated to its behavior in another region. This type of behavior may not be satisfactorily matched by any polynomial. This is attributed to the fact that the behavior of a polynomial everywhere is governed by its behavior in any small region. In such situations, it would be more appropriate to partition the domain of f Ž x . into several intervals and then use a different approximating polynomial, usually of low degree, in each subinterval. These polynomial segments can be joined in a smooth way, which leads to what is called a piecewise polynomial function. By definition, a spline function is a piecewise polynomial of degree n. The various polynomial segments Žall of degree n. are joined together at points called knots in such a way that the entire spline function is continuous and its first n y 1 derivatives are also continuous. Spline functions were first introduced by Schoenberg Ž1946.. 9.3.1. Properties of Spline Functions Let w a, b x can be interval, and let as ␶ 0 - ␶ 1 - ⭈⭈⭈ - ␶m - ␶mq1 s b be partition points in w a, b x. A spline function sŽ x . of degree n with knots at the points ␶ 1 , ␶ 2 , . . . , ␶m has the following properties: i. sŽ x . is a polynomial of degree not exceeding n on each subinterval w␶ iy1 , ␶ i x, 1 F i F m q 1. ii. sŽ x . has continuous derivatives up to order n y 1 on w a, b x. In particular, if n s 1, then the spline function is called a linear spline and can be represented as sŽ x . s

m

Ý ai < xy ␶ i < ,

is1

where a1 , a2 , . . . , a m are fixed numbers. We note that between any two knots, < xy ␶ i < , i s 1, 2, . . . , m, represents a straight-line segment. Thus the graph of sŽ x . is made up of straight-line segments joined at the knots.

419

APPROXIMATION BY SPLINE FUNCTIONS

We can obtain a linear spline that resembles Lagrange interpolation: Let ␪ 0 , ␪ 1 , . . . , ␪mq1 be given real numbers. For 1 F i F m, consider the functions

° xy ␶ y␶ ¢0, °0,

l Ž x . s~ ␶ 0

1

0

␶ 1 F xF ␶mq1 , xf w ␶ iy1 , ␶ iq1 x ,

xy ␶ iy1

~ ␶ y␶

li Ž x . s

i

␶ iy1 F x F ␶ i ,

,

iy1

␶ iq1 y x

¢␶ y ␶ , °0, xy ␶ ~ Ž x. s ¢␶ y ␶ , iq1

l mq1

␶ 0 F xF ␶ 1 ,

,

1

m

mq1

␶ i F xF ␶ iq1 ,

i

␶ 0 F xF ␶m , ␶m F xF ␶mq1 .

m

Then the linear spline sŽ x . s

mq1

Ý

␪i li Ž x .

Ž 9.21 .

is0

has the property that sŽ␶ i . s ␪ i , 0 F i F m q 1. It can be shown spline having this property is unique. Another special case is the cubic spline for n s 3. This is spline function in many applications. It can be represented as a i q bi xq c i x 2 q d i x 3 , ␶ iy1 F xF ␶ i , i s 1, 2, . . . , m q 1, such 1, 2, . . . , m,

that the linear a widely used sŽ x . s si Ž x . s that for i s

si Ž ␶ i . s siq1 Ž ␶ i . , sXi Ž ␶ i . s sXiq1 Ž ␶ i . , sYi Ž ␶ i . s sYiq1 Ž ␶ i . . In general, a spline of degree n with knots at ␶ 1 , ␶ 2 , . . . , ␶m is represented as sŽ x . s

m

Ý ei Ž x y ␶ i . qq p Ž x . ,

is1

n

Ž 9.22 .

420

APPROXIMATION OF FUNCTIONS

where e1 , e2 , . . . , e m are constants, pŽ x . is a polynomial of degree n, and n

Ž xy ␶ i . qs

½

n

Ž xy ␶ i . , 0,

xG ␶ i , xF ␶ i .

For an illustration, consider the cubic spline

½

sŽ x . s

a1 q b1 xq c1 x 2 q d1 x 3 ,

aF x F ␶ ,

a2 q b 2 xq c 2 x 2 q d 2 x 3 ,

␶ F xF b.

Here, sŽ x . along with its first and second derivatives must be continuous at xs ␶ . Therefore, we must have a1 q b1␶ q c1␶ 2 q d1␶ 3 s a2 q b 2␶ q c 2␶ 2 q d 2␶ 3 ,

Ž 9.23 .

b1 q 2 c1␶ q 3d1␶ 2 s b 2 q 2 c 2␶ q 3d 2␶ 2 ,

Ž 9.24 .

2 c1 q 6 d1␶ s 2 c 2 q 6 d 2␶ .

Ž 9.25 .

Equation Ž9.25. can be written as c 1 y c 2 s 3 Ž d 2 y d1 . ␶ .

Ž 9.26 .

From equations Ž9.24. and Ž9.26. we get b1 y b 2 q 3 Ž d 2 y d1 . ␶ 2 s 0.

Ž 9.27 .

Using now equations Ž9.26. and Ž9.27. in equation Ž9.23., we obtain a1 y a2 q 3Ž d1 y d 2 .␶ 3 q 3Ž d 2 y d1 .␶ 3 q Ž d1 y d 2 .␶ 3 s 0, or equivalently, a1 y a2 q Ž d1 y d 2 . ␶ 3 s 0.

Ž 9.28 .

We conclude that d 2 y d1 s

1

␶

3

Ž a1 y a2 . s

1 3␶

2

Ž b 2 y b1 . s

1 3␶

Ž c1 y c 2 . .

Ž 9.29 .

Let us now express sŽ x . in the form given by equation Ž9.22., that is, s Ž x . s e1 Ž x y ␶ . qq ␣ 0 q ␣ 1 xq ␣ 2 x 2 q ␣ 3 x 3 . 3

In this case,

␣ 0 s a1 ,

␣ 1 s b1 ,

␣ 2 s c1 ,

␣ 3 s d1 ,

421

APPROXIMATION BY SPLINE FUNCTIONS

and ye1␶ 3 q ␣ 0 s a2 ,

Ž 9.30 .

3e1␶ 2 q ␣ 1 s b 2 ,

Ž 9.31 .

y 3e1␶ q ␣ 2 s c 2 ,

Ž 9.32 .

e1 q ␣ 3 s d 2 .

Ž 9.33 .

In light of equation Ž9.29., equations Ž9.30. ᎐ Ž9.33. have a common solution for e1 given by e1 s d 2 y d1 s Ž1r3␶ .Ž c1 y c 2 . s Ž1r3␶ 2 .Ž b 2 y b1 . s Ž1r␶ 3 .Ž a1 y a2 .. 9.3.2. Error Bounds for Spline Approximation Let as ␶ 0 - ␶ 1 - ⭈⭈⭈ - ␶m - ␶mq1 s b be a partition of w a, b x. We recall that the linear spline sŽ x . given by formula Ž9.21. has the property that sŽ␶ i . s ␪ i , 0 F i F m q 1, where ␪ 0 , ␪ 1 , . . . , ␪mq1 are any given real numbers. In particular, if ␪ i is the value at ␶ i of a function f Ž x . defined on the interval w a, b x, then sŽ x . provides a spline approximation of f Ž x . over w a, b x which agrees with f Ž x . at ␶ 0 , ␶ 1 , . . . , ␶mq1 . If f Ž x . has continuous derivatives up to order 2 over w a, b x, then an upper bound on the error of approximation is given by Žsee De Boor, 1978, page 40. max aFxFb

ž

f Ž x . y s Ž x . F 18 max ⌬␶ i i

/

2

max aFxFb

f Ž2. Ž x . ,

where ⌬␶ i s ␶ iq1 y ␶ i , i s 0, 1, . . . , m. This error bound can be made small by reducing the value of max i ⌬␶ i . A more efficient and smoother spline approximation than the one provided by the linear spline is the commonly used cubic spline approximation. We recall that a cubic spline defined on w a, b x is a piecewise cubic polynomial that is twice continuously differentiable. Let f Ž x . be defined on w a, b x. There exists a unique cubic spline sŽ x . that satisfies the following interpolatory constraints: sŽ␶i . s f Ž␶i . ,

i s 0, 1, . . . , m q 1,

s⬘ Ž ␶ 0 . s f ⬘ Ž ␶ 0 . , s⬘ Ž ␶mq1 . s f ⬘ Ž ␶mq1 . ,

Ž 9.34 .

Žsee Prenter, 1975, Section 4.2.. If f Ž x . has continuous derivatives up to order 4 on w a, b x, then information on the error of approximation, which results from using a cubic spline, can be obtained from the following theorem, whose proof is given in Hall Ž1968.:

422

APPROXIMATION OF FUNCTIONS

Theorem 9.3.1. Let as ␶ 0 - ␶ 1 - ⭈⭈⭈ - ␶m - ␶mq1 s b be a partition of w a, b x. Let sŽ x . be a cubic spline associated with f Ž x . and satisfies the constraints described in Ž9.34.. If f Ž x . has continuous derivatives up to order 4 on w a, b x, then max aFxFb

ž

5 f Ž x . y s Ž x . F 384 max ⌬␶ i

i

/

4

max aFxFb

f Ž4. Ž x . ,

where ⌬␶ i s ␶ iq1 y ␶ i , i s 0, 1, . . . , m. Another advantage of cubic spline approximation is the fact that it can be used to approximate the first-order and second-order derivatives of f Ž x .. Hall and Meyer Ž1976. proved that if f Ž x . satisfies the conditions of Theorem 9.3.1, then max aFxFb

max aFxFb

ž

f ⬘ Ž x . y s⬘ Ž x . F 241 max ⌬␶ i

ž

i

f ⬙ Ž x . y s⬙ Ž x . F 38 max ⌬␶ i i

/

/

3

2

max aFxFb

max aFxFb

f Ž4. Ž x . , f Ž4. Ž x . .

Furthermore, the bounds concerning < f Ž x . y sŽ x .< and < f ⬘Ž x . y s⬘Ž x .< are best possible.

9.4. APPLICATIONS IN STATISTICS There is a wide variety of applications of polynomial approximation in statistics. In this section, we discuss the use of Lagrange interpolation in optimal design theory and the role of spline approximation in regression analysis. Other applications will be seen later in Chapter 10 ŽSection 10.9.. 9.4.1. Approximate Linearization of Nonlinear Models by Lagrange Interpolation We recall from Section 8.6 that a nonlinear model is one of the form y Ž x . s h Ž x, ␪ . q ⑀ ,

Ž 9.35 .

where x s Ž x 1 , x 2 , . . . , x k .⬘ is a vector of k input variables, ␪ s Ž ␪ 1 , ␪ 2 , . . . , ␪p .⬘ is a vector of p unknown parameters, ⑀ is a random error, and hŽx, ␪ . is a known function which is nonlinear in at least one element of ␪. We also recall that the choice of design for model Ž9.35., on the basis of the Box᎐Lucas criterion, depends on the values of the elements of ␪ that appear nonlinearly in the model. To overcome this undesirable design dependence problem, one possible approach is to construct an approximation

423

APPLICATIONS IN STATISTICS

to the mean response function hŽx, ␪ . with a Lagrange interpolating polynomial. This approximation can then be utilized to obtain a design for parameter estimation which does not depend on the parameter vector ␪. We shall restrict our consideration of model Ž9.35. to the case of a single input variable x. Let us suppose that the region of interest, R, is the interval w a, b x, and that ␪ belongs to a parameter space ⍀. We assume that: a. hŽ x, ␪ . has continuous partial derivatives up to order r q 1 with respect to x over w a, b x for all ␪ g ⍀, where r is such that r q 1 G p with p being the number of parameters in model Ž9.35., and is large enough so that 2

Ž r q 1. !

ž

by a 4

/

⭸ rq1 h Ž x, ␪ .

rq1

sup

⭸ x rq1

aFxFb

-␦

Ž 9.36 .

for all ␪ g ⍀, where ␦ is a small positive constant chosen appropriately so that the Lagrange interpolation of hŽ x, ␪ . achieves a certain accuracy. b. hŽ x, ␪ . has continuous first-order partial derivatives with respect to the elements of ␪. c. For any set of distinct points, x 0 , x 1 , . . . , x r , such that aF x 0 - x 1 - ⭈⭈⭈ - x r F b, where r is the integer defined in Ža., the p= Ž r q 1. matrix U Ž ␪ . s ⵱h Ž x 0 , ␪ . : ⵱h Ž x 1 , ␪ . : ⭈⭈⭈ :⵱h Ž x r , ␪ . is of rank p, where ⵱hŽ x i , ␪ . is the vector of partial derivatives of hŽ x i , ␪ . with respect to the elements of ␪ Ž i s 0, 1, . . . , r .. Let us now consider the points z 0 , z1 , . . . , z r , where z i is the ith Chebyshev point defined by formula Ž9.18.. Let pr Ž x, ␪ . denote the corresponding Lagrange interpolating polynomial for hŽ x, ␪ . over w a, b x, which utilizes the z i ’s as interpolation points. Then, by formula Ž9.14. we have pr Ž x, ␪ . s

r

Ý hŽ zi , ␪ . li Ž x . ,

Ž 9.37 .

is0

where l i Ž x . is a polynomial of degree r which can be obtained from formula Ž9.13. by substituting z i for a i Ž i s 0, 1, . . . , r .. By inequality Ž9.19., an upper bound on the error of approximating hŽ x, ␪ . with pr Ž x, ␪ . is given by sup aFxFb

h Ž x, ␪ . y pr Ž x, ␪ . F

2

Ž r q 1. !

ž

by a 4

/

rq1

sup aFxFb

⭸ rq1 h Ž x, ␪ . ⭸ x rq1

.

424

APPROXIMATION OF FUNCTIONS

However, by inequality Ž9.36., this upper bound is less than ␦ . We then have h Ž x, ␪ . y pr Ž x, ␪ . - ␦

sup

Ž 9.38 .

aFxFb

for all ␪ g ⍀. This provides the desired accuracy of approximation. On the basis of the above arguments, an approximate representation of model Ž9.35. is given by y Ž x . s pr Ž x, ␪ . q ⑀ .

Ž 9.39 .

Model Ž9.39. will now be utilized in place of hŽ x, ␪ . to construct an optimal design for estimating ␪. Let us now apply the Box᎐Lucas criterion described in Section 8.6 to approximate the mean response in model Ž9.39.. In this case, the matrix HŽ ␪ . wsee model Ž8.66.x is an n = p matrix whose Ž u, i .th element is ⭸ pr Ž x u , ␪ .r⭸␪ i , where x u is the design setting at the uth experimental run Ž u s 1, 2, . . . , n. and n is the number of experimental runs. From formula Ž9.37. we than have

⭸ pr Ž x u , ␪ . ⭸␪ i

r

s

Ý

js0

⭸ hŽ zj , ␪. ⭸␪ i

lj Ž xu . ,

i s 1, 2, . . . , p.

These equations can be written as ⵱pr Ž x u , ␪ . s U Ž ␪ . ␭ Ž x u . , where ␭ Ž x u . s w l 0 Ž x u ., l 1Ž x u ., . . . , l r Ž x u .x⬘ and UŽ ␪ . is the p= Ž r q 1. matrix U Ž ␪ . s ⵱h Ž z 0 , ␪ . : ⵱h Ž z1 , ␪ . : ⭈⭈⭈ :⵱h Ž z r , ␪ . . By assumption Žc., UŽ ␪ . is of rank p. The matrix HŽ ␪ . is therefore of the form H Ž ␪ . s ⌳U⬘ Ž ␪ . , where ⌳⬘ s ␭ Ž x 1 . : ␭ Ž x 2 . : ⭈⭈⭈ : ␭ Ž x n . . Thus H⬘ Ž ␪ . H Ž ␪ . s U Ž ␪ . ⌳⬘⌳U⬘ Ž ␪ . .

Ž 9.40 .

If n G r q 1 and at least r q 1 of the design points Žthat is, x 1 , x 2 , . . . , x n . are distinct, then ⌳⬘⌳ is a nonsingular matrix. To show this, it is sufficient to prove that ⌳ is of full column rank r q 1. If not, then there must exist constants ␣ 0 , ␣ 1 , . . . , ␣ r , not all equal to zero, such that r

Ý ␣ i l i Ž x u . s 0,

is0

u s 1, 2, . . . , n.

425

APPLICATIONS IN STATISTICS

This indicates that the r th degree polynomial Ý ris0 ␣ i l i Ž x . has n roots, namely, x 1 , x 2 , . . . , x n . This is not possible, because n G r q 1 and at least r q 1 of the x u’s Ž u s 1, 2, . . . , n. are distinct Ža polynomial of degree r has at most r distinct roots.. This contradiction implies that ⌳ is of full column rank and ⌳⬘⌳ is therefore nonsingular. Applying the Box᎐Lucas design criterion to the approximating model Ž9.37. amounts to finding the design settings that maximize detwH⬘Ž ␪ .HŽ ␪ .x. From formula Ž9.40. we have det H⬘ Ž ␪ . H Ž ␪ . s det U Ž ␪ . ⌳⬘⌳U⬘ Ž ␪ . .

Ž 9.41 .

We note that the matrix ⌳⬘⌳ s Ý nus1 ␭ Ž x u . ␭⬘Ž x u . depends only on the design settings. Let ␯min Ž x 1 , x 2 , . . . , x n . and ␯max Ž x 1 , x 2 , . . . , x n . denote, respectively, the smallest and the largest eigenvalue of ⌳⬘⌳. These eigenvalues are positive, since ⌳⬘⌳ is positive definite by the fact that ⌳⬘⌳ is nonsingular, as was shown earlier. From formula Ž9.41. we conclude that p det U Ž ␪ . U⬘ Ž ␪ . ␯min Ž x 1 , x 2 , . . . , x n . F det H⬘ Ž ␪ . H Ž ␪ . p F det U Ž ␪ . U⬘ Ž ␪ . ␯max Ž x1 , x 2 , . . . , x n . .

Ž 9.42 . This double inequality follows from the fact that the matrices ␯ m a x Ž x 1 , x 2 , . . . , x n . U Ž ␪ . U ⬘ Ž ␪ . y H ⬘ Ž ␪ . H Ž ␪ . an d H ⬘ Ž ␪ . H Ž ␪ . y ␯min Ž x 1 , x 2 , . . . , x n .UŽ ␪ .U⬘Ž ␪ . are positive semidefinite. An application of Theorem 2.3.19Ž1. to these matrices results in the double inequality Ž9.42. Žwhy?.. Note that the determinant of UŽ ␪ .U⬘Ž ␪ . is not zero, since UŽ ␪ .U⬘Ž ␪ ., which is of order p= p, is of rank p by assumption Ž c .. Now, from the double inequality Ž9.42. we deduce that there exists a number ␥ , 0 F ␥ F 1, such that p p det H⬘ Ž ␪ . H Ž ␪ . s ␥␯min Ž x 1 , x 2 , . . . , x n . q Ž 1 y ␥ . ␯max Ž x1 , x 2 , . . . , x n .

= det U Ž ␪ . U⬘ Ž ␪ . . If ␥ is integrated out, we obtain

H0 det H⬘ Ž ␪ . H Ž ␪ . 1

p p d␥ s 12 ␯min Ž x 1 , x 2 , . . . , x n . q ␯max Ž x1 , x 2 , . . . , x n .

= det U Ž ␪ . U⬘ Ž ␪ . . Consequently, to construct an optimal design we can consider finding x 1 , x 2 , . . . , x n that maximize the function p p ␺ Ž x 1 , x 2 , . . . , x n . s 12 ␯min Ž x 1 , x 2 , . . . , x n . q ␯max Ž x 1 , x 2 , . . . , x n . . Ž 9.43 .

426

APPROXIMATION OF FUNCTIONS

This is a modified version of the Box᎐Lucas criterion. Its advantage is that the optimal design is free of ␪. We therefore call such a design a parameterfree design. The maximization of ␺ Ž x 1 , x 2 , . . . , x n . can be conveniently carried out by using a FORTRAN program written by Conlon Ž1991., which is based on Price’s Ž1977. controlled random search procedure. EXAMPLE 9.4.1. Let us consider the nonlinear model used by Box and Lucas Ž1959. of a consecutive first-order chemical reaction in which a raw material A reacts to form a product B, which in turn decomposes to form substance C. After time x has elapsed, the mean yield of the intermediate product B is given by h Ž x, ␪ . s

␪1 ␪1 y ␪2

Ž ey␪ 2 x y ey␪ 1 x . ,

where ␪ 1 and ␪ 2 are the rate constants for the reactions A ™ B and B ™ C, respectively. Suppose that the region of interest R is the interval w0, 10x. Let the parameter space ⍀ be such that 0 F ␪ 1 F 1, 0 F ␪ 2 F 1. It can be verified that

⭸ rq1 h Ž x, ␪ . ⭸x

rq1

s Ž y1 .

rq1

␪1 ␪1 y␪2

Ž ␪ 2rq1 ey␪

2

x

y ␪ 1rq1 ey␪ 1 x . .

Let us consider the function ␻ Ž x, ␾ . s ␾ rq1 ey␾ x. By the mean value theorem ŽTheorem 4.2.2.,

␪ 2rq1 ey␪ 2 x y ␪ 1rq1 ey␪ 1 x s Ž ␪ 2 y ␪ 1 .

⭸ w Ž x, ␪# . ⭸␾

,

where ⭸ w Ž x, ␪#.r⭸␾ is the partial derivative of w Ž x, ␾ . with respect to ␾ evaluated at ␪#, and where ␪# is between ␪ 1 and ␪ 2 . Thus r y␪ # x rq1 ␪ 2rq1 ey␪ 2 x y ␪ 1rq1 ey␪ 1 x s Ž ␪ 2 y ␪ 1 . Ž r q 1 . ␪#e y ␪# xey␪# x .

Hence,

sup 0FxF10

⭸ rq1 h Ž x, ␪ . ⭸ x rq1

F ␪ 1 ␪#r sup

ey␪ # x < r q 1 y x ␪# <

0FxF10

F sup < r q 1 y x ␪# < . 0FxF10

427

APPLICATIONS IN STATISTICS

However, < r q 1 y x ␪# < s

½

r q 1 y x ␪# yr y 1 q x ␪#

if r q 1 G x ␪#, if r q 1 - x ␪#.

Since 0 F x ␪# F 10, then sup < r q 1 y x ␪# < F max Ž r q 1, 9 y r . . 0FxF10

We then have

⭸ rq1 h Ž x, ␪ .

sup 0FxF10

⭸ x rq1

F max Ž r q 1, 9 y r . .

By inequality Ž9.36., the integer r is determined such that 2

Ž r q 1. !

ž / 10 4

rq1

max Ž r q 1, 9 y r . - ␦ .

Ž 9.44 .

If we choose ␦ s 0.053, for example, then it can be verified that the smallest positive integer that satisfies inequality Ž9.44. is r s 9. The Chebyshev points in formula Ž9.18. that correspond to this value of r are given in Table 9.2. On choosing n, the number of design points, to be equal to r q 1 s 10, where all ten design points are distinct, the matrix ⌳ in formula Ž9.40. will be nonsingular of order 10 = 10. Using Conlon’s Ž1991. FORTRAN program for the maximization of the function ␺ in formula Ž9.43. with ps 2, it can be shown that the maximum value of ␺ is 17.457. The corresponding optimal values of x 1 , x 2 , . . . , x 10 are given in Table 9.2. Table 9.2. Chebyshev Points and Optimal Design Points for Example 9.4.1 Chebyshev Points

Optimal Design Points

9.938 9.455 8.536 7.270 5.782 4.218 2.730 1.464 0.545 0.062

9.989 9.984 9.983 9.966 9.542 7.044 6.078 4.038 1.381 0.692

428

APPROXIMATION OF FUNCTIONS

9.4.2. Splines in Statistics There is a broad variety of work on splines in statistics. Spline functions are quite suited in practical applications involving data that arise from the physical world rather than the mathematical world. It is therefore only natural that splines have many useful applications in statistics. Some of these applications will be discussed in this section. 9.4.2.1. The Use of Cubic Splines in Regression Let us consider fitting the model ysg Ž x. q⑀ ,

Ž 9.45 .

where g Ž x . is the mean response at x and ⑀ is a random error. Suppose that the domain of x is divided into a set of m q 1 intervals by the points ␶ 0 - ␶ 1 - ⭈⭈⭈ - ␶m - ␶mq1 such that on the ith interval Ž i s 1, 2, . . . , m q 1., g Ž x . is represented by the cubic spline si Ž x . s a i q bi xq c i x 2 q d i x 3 ,

␶ iy1 F xF ␶ i .

Ž 9.46 .

As was seen earlier in Section 9.3.1, the parameters a i , bi , c i , d i Ž i s 1, 2, . . . , m q 1. are subject to the following continuity restrictions: a i q bi␶ i q c i␶ i2 q d i␶ i3 s a iq1 q biq1␶ i q c iq1␶ i2 q d iq1␶ i3 ,

Ž 9.47 .

that is, si Ž␶ i . s siq1 Ž␶ i ., i s 1, 2, . . . , m; bi q 2 c i␶ i q 3d i␶ i2 s biq1 q 2 c iq1␶ i q 3d iq1␶ i2 ,

Ž 9.48 .

that is, sXi Ž␶ i . s sXiq1 Ž␶ i ., i s 1, 2, . . . , m; and 2 c i q 6 d i␶ i s 2 c iq1 q 6 d iq1␶ i ,

Ž 9.49 .

that is, sYi Ž␶ i . s sYiq1 Ž␶ i ., i s 1, 2, . . . , m. The number of unknown parameters in model Ž9.45. is therefore equal to 4Ž m q 1.. The continuity restrictions Ž9.47. ᎐ Ž9.49. reduce the dimensionality of the parameter space to m q 4. However, only m q 2 parameters can be estimated. This is because the spline method does not estimate the parameters of the si ’s directly, but estimates the ordinates of the si ’s at the points ␶ 0 , ␶ 1 , . . . , ␶mq1 , that is, s1Ž␶ 0 . and si Ž␶ i ., i s 1, 2, . . . , m q 1. Two additional restrictions are therefore needed. These are chosen to be of the form Žsee Poirier, 1973, page 516; Buse and Lim, 1977, page 64.: sY1 Ž ␶ 0 . s ␲ 0 sY1 Ž ␶ 1 . , or 2 c1 q 6 d1␶ 0 s ␲ 0 Ž 2 c1 q 6 d1␶ 1 . ,

Ž 9.50 .

429

APPLICATIONS IN STATISTICS

and sYmq1 Ž ␶mq1 . s ␲mq1 sYmq1 Ž ␶m . , or 2 c mq1 q 6 d mq1 ␶mq1 s ␲mq1 Ž 2 c mq1 q 6 d mq1 ␶m . ,

Ž 9.51 .

where ␲ 0 and ␲mq1 are known. Let y 1 , y 2 , . . . , yn be n observations on the response y, where n ) m q 2, such that n i observations are taken in the ith interval w␶ iy1 , ␶ i x, i s 1, 2, . . . , m q 1. Thus n s Ý mq1 is1 n i . If yi1 , yi2 , . . . , yi n i are the observations in the ith interval Ž i s 1, 2, . . . , m q 1., then from model Ž9.45. we have yi j s g Ž x i j . q ⑀ i j ,

i s 1, 2, . . . , m q 1; j s 1, 2, . . . , n i ,

Ž 9.52 .

where x i j is the setting of x for which y s yi j , and the ⑀ i j ’s are distributed independently with means equal to zero and a common variance ␴ 2 . The estimation of the parameters of model Ž9.45. is then reduced to a restricted least-squares problem with formulas Ž9.47. ᎐ Ž9.51. representing linear restrictions on the 4Ž m q 1. parameters of the model wsee, for example, Searle Ž1971, Section 3.6., for a discussion concerning least-squares estimation under linear restrictions on the fitted model’s parameters x. Using matrix notation, model Ž9.52. and the linear restrictions Ž9.47. ᎐ Ž9.51. can be expressed as

Ž 9.53 .

y s X␤ q ⑀ ,

Ž 9.54 .

C␤ s ␦ ,

X .⬘ with yi s Ž yi1 , yi2 , . . . , yi n .⬘, i s 1, 2, . . . , m q 1, X where y s Žy1X : y2X : ⭈⭈⭈ : ymq1 i Ž . s Diga X 1 , X 2 , . . . , X mq1 is a block-diagonal matrix of order n = w4Ž m q 1.x with X i being a matrix of order n i = 4 whose jth row is of the form Ž1, x i j , x i2j , x i3j ., j s 1, 2, . . . , n i ; i s 1, 2, . . . , m q 1; ␤ s Ž␤X1: ␤X2 : ⭈⭈⭈ : ␤Xmq1 .⬘ with ␤ i s Ž a i , bi , c i , d i .⬘, i s 1, 2, . . . , m q 1; and ⑀ s Ž ⑀X1: ⑀X2 : ⭈⭈⭈ : ⑀Xmq1 .⬘, where ⑀ j is the vector of random errors associated with the observations in the ith interval, i s 1, 2, . . . , m q 1. Furthermore, C s wCX0 : CX1: CX2 : CX3 x⬘, where, for l s 0, 1, 2,

yeX1 l 0⬘ . Cl s . . 0⬘

eX1 l yeX2 l . . . 0⬘

0⬘ eX2 l . . . 0⬘

⭈⭈⭈ ⭈⭈⭈ ⭈⭈⭈

0⬘ 0⬘ . . . yeXm l

0⬘ 0⬘ . . . eXm l

430

APPROXIMATION OF FUNCTIONS

is a matrix of order m = w4Ž m q 1.x such that eXi0 s Ž1, ␶ i , ␶ i2 , ␶ i3 ., eXi1 s Ž0, 1, 2␶ i , 3␶ i2 ., eXi2 s Ž0, 0, 2, 6␶ i ., i s 1, 2, . . . , m, and C3 s

0

0

2 Ž␲ 0 y 1.

6 Ž ␲ 0␶ 1 y ␶ 0 .

⭈⭈⭈

0

0

0

0

0

0

0

0

⭈⭈⭈

0

0

2 Ž ␲mq1 y 1 .

6 Ž ␲mq1 ␶m y ␶mq1 .

is a 2 = w4Ž m q 1.x matrix. Finally, ␦ s Ž ␦X0 : ␦X1: ␦X2 : ␦X3 .⬘ s 0, where the partitioning of ␦ into ␦ 0 , ␦ 1 , ␦ 2 , and ␦ 3 conforms to that of C. Consequently, and on the basis of formula Ž103. in Searle Ž1971, page 113., the least-squares estimator of ␤ for model Ž9.53. under the restriction described by formula Ž9.54. is given by

ˆrs␤ ˆ y Ž X⬘X . ␤

y1

C⬘ C Ž X⬘X .

y1

ˆ y Ž X⬘X . s␤

y1

C⬘ C Ž X⬘X .

y1

y1

C⬘ y1

C⬘

Ž C␤ˆ y ␦ . ˆ, C␤

ˆ s ŽX⬘X.y1 X⬘y is the ordinary least-squares estimator of ␤. where ␤ This estimation procedure, which was developed by Buse and Lim Ž1977., demonstrates that the fitting of a cubic spline regression model can be reduced to a restricted least-squares problem. Buse and Lim presented a numerical example based on Indianapolis 500 race data over the period Ž1911᎐1971. to illustrate the implementation of their procedure. Other papers of interest in the area of regression splines include those of Poirier Ž1973. and Gallant and Fuller Ž1973.. The paper by Poirier discusses the basic theory of cubic regression splines from an economic point of view. In the paper by Gallant and Fuller, the knots are treated as unknown parameters rather than being fixed. Thus in their procedure, the knots must be estimated, which causes the estimation process to become nonlinear. 9.4.2.2. Designs for Fitting Spline Models A number of papers have addressed the problem of finding a design to estimate the parameters of model Ž9.45., where g Ž x . is represented by a spline function. We shall make a brief reference to some of these papers. Agarwal and Studden Ž1978. considered a representation of g Ž x . over 0 F xF 1 by a linear spline sŽ x ., which has the form given by Ž9.21.. Here, g ⬙ Ž x . is assumed to be continuous. If we recall, the ␪ i coefficients in formula Ž9.21. are the values of s at ␶ 0 , ␶ 1 , . . . , ␶mq1 . Let x 1 , x 2 , . . . , x r be r design points in w0, 1x. Let yi denote the average of n i observations taken at x i Ž i s 1, 2, . . . , r .. The vector ␪ s Ž ␪ 0 , ␪ 1 , . . . , ␪mq1 .⬘ can therefore be estimated by

ˆ␪ s Ay,

Ž 9.55 .

where y s Ž y 1 , y 2 , . . . , yr .⬘ and A is an Ž m q 2. = r matrix. Hence, an estimate of g Ž x . is given by

ˆg Ž x . s l⬘ Ž x . ˆ␪ s l⬘ Ž x . Ay, where lŽ x . s w l 0 Ž x ., l 1Ž x ., . . . , l mq1 Ž x .x⬘.

Ž 9.56 .

431

APPLICATIONS IN STATISTICS

Now, E Žˆ ␪ . s Ag r , where g r s w g Ž x 1 ., g Ž x 2 ., . . . , g Ž x r .x⬘. Thus E w ˆ g Ž x .x s l⬘Ž x .Ag r , and the variance of ˆ g Ž x . is 2

Var ˆ g Ž x . s E l⬘ Ž x . ˆ ␪ y l⬘ Ž x . Ag r s

␴2 n

l⬘ Ž x . ADy1A⬘l Ž x . ,

where D is an r = r diagonal matrix with diagonal elements n1rn, n 2rn, . . . , n rrn. The mean squared error of ˆ g Ž x . is the variance plus the squared bias of ˆ g Ž x .. It follows that the integrated mean squared error ŽIMSE. of ˆ g Ž x . Žsee Section 8.4.3. is Js

n␻

␴

2

H0 Var 1

ˆg Ž x . dxq

n␻

␴2

H0 Bias 1

2

ˆg Ž x . dx

s V q B, where ␻ s Ž H01 dx .y1 s 12 , and Bias 2 ˆ g Ž x . s g Ž x . y l⬘ Ž x . Ag r . 2

Thus Js s

1

H l⬘ Ž x . AD 2 0 1 2

1

A⬘l Ž x . dxq

y1

tr Ž ADy1A⬘M . q

n 2␴

H0

1

2

n 2␴ 2

H0

1

g Ž x . y l⬘ Ž x . Ag r

g Ž x . y l⬘ Ž x . Ag r

2

2

dx

dx,

where M s H01 lŽ x .l⬘Ž x . dx. Agarwal and Studden Ž1978. proposed to minimize J with respect to Ži. the design Žthat is, x 1 , x 2 , . . . , x r as well as n1 , n 2 , . . . , n r ., Žii. the matrix A, and Žiii. the knots ␶ 1 , ␶ 2 , . . . , ␶m , assuming that g is known. Park Ž1978. adopted the D-optimality criterion Žsee Section 8.5. for the choice of design when g Ž x . is represented by a spline of the form given by formula Ž9.22. with only one intermediate knot. Draper, Guttman, and Lipow Ž1977. extended the design criterion based on the minimization of the average squared bias B Žsee Section 8.4.3. to situations involving spline models. In particular, they considered fitting first-order or second-order models when the true mean response is of the second order or the third order, respectively. 9.4.2.3. Other Applications of Splines in Statistics Spline functions have many other useful applications in both theoretical and applied statistical research. For example, splines are used in nonparametric

432

APPROXIMATION OF FUNCTIONS

regression and data smoothing, nonparametric density estimation, and time series analysis. They are also utilized in the analysis of response curves in agriculture and economics. The review articles by Wegman and Wright Ž1983. and Ramsay Ž1988. contain many references on the various uses of splines in statistics Žsee also the article by Smith, 1979.. An overview of the role of splines in regression analysis is given in Eubank Ž1984..

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Agarwal, G. G., and W. J. Studden Ž1978., ‘‘Asymptotic design and estimation using linear splines.’’ Comm. Statist. Simulation Comput., 7, 309᎐319. Box, G. E. P., and H. L. Lucas Ž1959.. ‘‘Design of experiments in nonlinear situations.’’ Biometrika, 46, 77᎐90. Buse, A., and L. Lim Ž1977.. ‘‘Cubic splines as a special case of restricted least squares.’’ J. Amer. Statist. Assoc., 72, 64᎐68. Cheney, E. W. Ž1982.. Introduction to Approximation Theory, 2nd ed. Chelsea, New York. ŽThe Weierstrass approximation theorem and Lagrange interpolation are covered in Chap. 3; least-squares approximation is discussed in Chap. 4.. Conlon, M. Ž1991.. ‘‘The controlled random search procedure for function optimization.’’ Personal communication. ŽThis is a FORTRAN file for implementing Price’s controlled random search procedure. . Cornish, E. A., and R. A. Fisher Ž1937.. ‘‘Moments and cumulants in the specification of distribution.’’ Re®. Internat. Statist. Inst., 5, 307᎐320. Cramer, ´ H. Ž1946.. Mathematical Methods of Statistics. Princeton University Press, Princeton. ŽThis classic book provides the mathematical foundation of statistics. Chap. 17 is a good source for approximation of density functions. . Davis, P. J. Ž1975.. Interpolation and Approximation. Dover, New York. ŽChaps. 2, 3, 6, 8, and 10 are relevant to the material on Lagrange interpolation, least-squares approximation, and orthogonal polynomials.. De Boor, C. Ž1978.. A Practical Guide to Splines. Springer-Verlag, New York. ŽChaps. 1 and 2 provide a good coverage of Lagrange interpolation, particularly with regard to the use of Chebyshev points. Chap. 4 discusses cubic spline approximation.. Draper, N. R., I. Guttman, and P. Lipow Ž1977.. ‘‘All-bias designs for spline functions joined at the axes.’’ J. Amer. Statist. Assoc., 72, 424᎐429. Eubank, R. L. Ž1984.. ‘‘Approximate regression models and splines.’’ Comm. Statist. Theory Methods, 13, 433᎐484. Gallant, A. R., and W. A. Fuller Ž1973.. ‘‘Fitting segmented polynomial regression models whose join points have to be estimated.’’ J. Amer. Statist. Assoc., 68, 144᎐147. Graybill, F. A. Ž1983.. Matrices with Applications in Statistics, 2nd ed. Wadsworth, Belmont, California. Hall, C. A. Ž1968.. ‘‘On error bounds for spline interpolation.’’ J. Approx. Theory, 1, 209᎐218.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

433

Hall, C. A., and W. W. Meyer Ž1976.. ‘‘Optimal error bounds for cubic spline interpolation.’’ J. Approx. Theory, 16, 105᎐122. Harris, B. Ž1966.. Theory of Probability. Addison-Wesley, Reading, Massachusetts. Johnson, N. L., and S. Kotz Ž1970.. Continuous Uni®ariate Distributionsᎏ1. Houghton Mifflin, Boston. ŽChap. 12 contains a good discussion concerning the Cornish᎐Fisher expansion of percentage points.. Kendall, M. G., and A. Stuart Ž1977.. The Ad®anced Theory of Statistics, Vol. 1, 4th ed. Macmillan, New York. ŽThis classic book provides a good source for learning about the Gram᎐Charlier series of type A and the Cornish᎐Fisher expansion. . Lancaster, P., and K. Salkauskas Ž1986.. Cur®e and Surface Fitting. Academic Press, London. ŽThis book covers the foundations and major features of several basic methods for curve and surface fitting that are currently in use.. Park, S. H. Ž1978.. ‘‘Experimental designs for fitting segmented polynomial regression models.’’ Technometrics, 20, 151᎐154. Poirier, D. J. Ž1973.. ‘‘Piecewise regression using cubic splines.’’ J. Amer. Statist. Assoc., 68, 515᎐524. Powell, M. J. D. Ž1967.. ‘‘On the maximum errors of polynomial approximation defined by interpolation and by least squares criteria.’’ Comput. J., 9, 404᎐407. Prenter, P. M. Ž1975.. Splines and Variational Methods. Wiley, New York. ŽLagrange interpolation is covered in Chap. 2; cubic splines are discussed in Chap. 4. An interesting feature of this book is its coverage of polynomial approximation of a function of several variables. . Price, W. L. Ž1977.. ‘‘A controlled random search procedure for global optimization.’’ Comput. J., 20, 367᎐370. Ramsay, J. O. Ž1988.. ‘‘Monotone regression splines in action.’’ Statist. Sci., 3, 425᎐461. Rice, J. R. Ž1969.. The Approximation of Functions, Vol. 2. Addison-Wesley, Reading, Massachusetts. ŽApproximation by spline functions is presented in Chap. 10.. Rivlin, T. J. Ž1969.. An Introduction to the Approximation of Functions. Dover, New York. ŽThis book provides an introduction to some of the most significant methods of approximation of functions by polynomials. Spline approximation is also discussed. . Schoenberg, I. J. Ž1946.. ‘‘Contributions to the problem of approximation of equidistant data by analytic functions.’’ Quart. Appl. Math., 4, Part A, 45᎐99; Part B, 112᎐141. Searle, S. R. Ž1971.. Linear Models. Wiley, New York. Smith, P. L. Ž1979.. ‘‘Splines as a useful and convenient statistical tool.’’ Amer. Statist., 33, 57᎐62. Szidarovszky, F., and S. Yakowitz Ž1978.. Principles and Procedures of Numerical Analysis. Plenum Press, New York. ŽChap. 2 provides a brief introduction to approximation and interpolation of functions. . Wegman, E. J., and I. W. Wright Ž1983.. ‘‘Splines in statistics.’’ J. Amer. Statist. Assoc., 78, 351᎐365. Wold, S. Ž1974.. ‘‘Spline functions in data analysis.’’ Technometrics, 16, 1᎐11.

434

APPROXIMATION OF FUNCTIONS

EXERCISES In Mathematics 9.1. Let f Ž x . be a function with a continuous derivative on w0, 1x, and let bnŽ x . be the nth degree Bernstein approximating polynomial of f. Then, for some constant c and for all n, f Ž x . y bn Ž x . F

sup 0FxF1

c 1r2

n

.

9.2. Prove Lemma 9.1.1. 9.3. Prove Lemma 9.1.2. 9.4. Show that for every interval wya, ax there is a sequence of polynomials pnŽ x . such that pnŽ0. s 0 and lim n™⬁ pnŽ x . s < x < uniformly on wya, ax. 9.5. Suppose that f Ž x . is continuous on w0, 1x and that

H0 f Ž x . x 1

n

dxs 0,

n s 0, 1, 2, . . . .

Show that f Ž x . s 0 on w0, 1x. w Hint: H01 f Ž x . pnŽ x . dxs 0, where pnŽ x . is any polynomial of degree n.x 9.6. Suppose that the function f Ž x . has n q 1 continuous derivatives on w a, b x. Let as a0 - a1 - ⭈⭈⭈ - a n s b be n q 1 points in w a, b x. Then sup aFxFb

f Ž x . y pŽ x . F

␶nq1 h nq1 4 Ž n q 1.

,

where pŽ x . is the Lagrange polynomial defined by formula Ž9.14., ␶nq1 s sup aF x F b < f Ž nq1. Ž x .< , and h s maxŽ a iq1 y a i ., i s 0, 1, . . . , n y 1. n Ž w Hint: Show that < Ł is0 x y a i .< F n!Ž h nq1 r4..x 9.7. Apply Lagrange interpolation to approximate the function f Ž x . s log x over the interval w3.50, 3.80x using a0 s 3.50, a1 s 3.60, a2 s 3.70, and a3 s 3.80 as interpolation points. Compute an upper bound on the error of approximation. 9.8. Let as ␶ 0 - ␶ 1 - ⭈⭈⭈ - ␶n s b be a partition of w a, b x. Suppose that f Ž x . has continuous derivatives up to order 2 over w a, b x. Consider a

435

EXERCISES

cubic spline sŽ x . that satisfies sŽ␶i . s f Ž␶i . ,

i s 0, 1, . . . , n,

s⬘ Ž a . s f ⬘ Ž a . , s⬘ Ž b . s f ⬘ Ž b . . Show that

Ha

b

f⬙ Ž x.

2

dxG

Ha

b

s⬙ Ž x .

2

dx.

9.9. Determine the cubic spline approximation of the function f Ž x . s cosŽ2␲ x . over the interval w0, ␲ x using five evenly spaced knots. Give an upper bound on the error approximation. In Statistics 9.10. Consider the nonlinear model y Ž x . s h Ž x, ␪ . q ⑀ , where h Ž x, ␪ . s ␪ 1 Ž 1 y ␪ 2 ey␪ 3 x . , such that 0 F ␪ 1 F 50, 0 F ␪ 2 F 1, 0 F ␪ 3 F 1. Obtain a Lagrange interpolating polynomial that approximates the mean response function hŽ x, ␪ . over the region w0, 8x with an error not exceeding ␦ s 0.05. 9.11. Consider the nonlinear model y s ␣ q Ž 0.49 y ␣ . exp y␤ Ž xy 8 . q ⑀ , where ⑀ is a random error with a zero mean and a variance ␴ 2 . Suppose that the region of interest is the interval w10, 40x, and that the parameter space ⍀ is such that 0.36 F ␣ F 0.41, 0.06 F ␤ F 0.16. Let sŽ x . be the cubic spline that approximates the mean response, that is, ␩ Ž x, ␣ , ␤ . s ␣ q Ž0.49 y ␣ . expwy␤ Ž xy 8.x, over w10, 40x. Determine the number of knots needed so that max 10FxF40

for all Ž ␣ , ␤ . g ⍀.

␩ Ž x, ␣ , ␤ . y s Ž x . - 0.001

436

APPROXIMATION OF FUNCTIONS

9.12. Consider fitting the spline model y s ␤ 0 q ␤ 1 xq ␤ 2 Ž xy ␣ . qq ⑀ 2

over the interval wy1, 1x, where ␣ is a known constant, y1 F ␣ F 1. A three-point design consisting of x 1 , x 2 , x 3 with y1 F x 1 - ␣ F x 2 - x 3 F 1 is used to fit the model. Using matrix notation, the model is written as y s X␤ q ⑀ . where X is the matrix 1

x1

0

Xs 1

x2

Ž x2 y ␣ .

2

x3

Ž x3 y ␣ .

2

1

,

and ␤ s Ž ␤ 0 , ␤ 1 , ␤ 2 .⬘. Determine x 1 , x 2 , x 3 so that the design is Doptimal, that is, it maximizes the determinant of X⬘X. w Note: See Park Ž1978..x

CHAPTER 10

Orthogonal Polynomials

The subject of orthogonal polynomials can be traced back to the work of the French mathematician Adrien-Marie Legendre Ž1752᎐1833. on planetary motion. These polynomials have important applications in physics, quantum mechanics, mathematical statistics, and other areas in mathematics. This chapter provides an exposition of the properties of orthogonal polynomials. Emphasis will be placed on Legendre, Chebyshev, Jacobi, Laguerre, and Hermite polynomials. In addition, applications of these polynomials in statistics will be discussed in Section 10.9.

10.1. INTRODUCTION Suppose that f Ž x . and g Ž x . are two continuous functions on w a, b x. Let w Ž x . be a positive function that is Riemann integrable on w a, b x. The dot product of f Ž x . and g Ž x . with respect to w Ž x ., which is denoted by Ž f ⭈ g .␻ , is defined as

Ž f ⭈ g . ␻ s H f Ž x . g Ž x . w Ž x . dx. b

a

The norm of f Ž x . with respect to w Ž x ., denoted by 5 f 5 ␻ , is defined as 5 f 5 ␻ s w Hab f 2 Ž x . w Ž x . dx x1r2 . The functions f Ž x . and g Ž x . are said to be orthogonal wwith respect to w Ž x .x if Ž f ⭈ g .␻ s 0. Furthermore, a sequence
f nŽ x .4⬁ns0 of continuous functions defined on w a, b x are said to be orthogonal with respect to w Ž x . if Ž f m ⭈ f n .␻ s 0 for m / n. If, in addition, 5 f n 5 ␻ s 1 for all n, then the functions f nŽ x ., n s 0, 1, 2, . . . , are called orthonormal. In particular, if S s
pnŽ x .4⬁ns0 is a sequence of polynomials such that Ž pn ⭈ pm .␻ s 0 for all m / n, then S forms a sequence of polynomials orthogonal with respect to w Ž x .. A sequence of orthogonal polynomials can be constructed on the basis of the following theorem: 437

438

ORTHOGONAL POLYNOMIALS

Theorem 10.1.1. The polynomials
pnŽ x .4⬁ns0 which are defined according to the following recurrence relation are orthogonal: p 0 Ž x . s 1, p1 Ž x . s xy

Ž xp0 ⭈ p 0 . ␻ 2 p0 ␻

s xy

Ž x⭈ 1 . ␻

,

2 ␻

1

Ž 10.1 .

. . . pn Ž x . s Ž xy a n . pny1 Ž x . y bn pny2 Ž x . ,

n s 2, 3, . . . ,

where an s

bn s

Ž xpny1 ⭈ pny1 . ␻

Ž 10.2 .

2 ␻

pny1

Ž xpny1 ⭈ pny2 . ␻

Ž 10.3 .

2 ␻

pny2

Proof. We show by mathematical induction on n that Ž pn ⭈ pi .␻ s 0 for i - n. For n s 1,

Ž p1 ⭈ p 0 . ␻ s H xy b

a

Ž x⭈ 1 . ␻ 1

w Ž x . dx

2 ␻

s Ž x⭈ 1 . ␻ y Ž x⭈ 1 . ␻

1

2 ␻

1

2 ␻

s 0. Now, suppose that the assertion is true for n y 1 Ž n G 2.. To show that it is true for n. We have that

Ž pn ⭈ pi . ␻ s H Ž x y an . pny1 Ž x . y bn pny2 Ž x . pi Ž x . w Ž x . dx b

a

s Ž xpny1 ⭈ pi . ␻ y a n Ž pny1 ⭈ pi . ␻ y bn Ž pny2 ⭈ pi . ␻ . Thus, for i s n y 1,

Ž pn ⭈ pi . ␻ s Ž xpny1 ⭈ pny1 . ␻ y an pny1 s 0,

2 ␻ y bn

Ž pny2 ⭈ pny1 . ␻

439

INTRODUCTION

by the definition of a n in Ž10.2. and the fact that Ž pny2 ⭈ pny1 .␻ s Ž pny1 ⭈ pny2 .␻ s 0. Similarly, for i s n y 2,

Ž pn ⭈ pi . ␻ s Ž xpny1 ⭈ pny2 . ␻ y an Ž pny1 ⭈ pny2 . ␻ y bn Ž pny2 ⭈ pny2 . ␻ s Ž xpny1 ⭈ pny2 . ␻ y bn pny2

2 ␻

by Ž 10.3 . .

s 0,

Finally, for i - n y 2, we have

Ž pn ⭈ pi . ␻ s Ž xpny1 ⭈ pi . ␻ y an Ž pny1 ⭈ pi . ␻ y bn Ž pny2 ⭈ pi . ␻ s

Ha xp b

ny1

Ž x . pi Ž x . w Ž x . dx.

Ž 10.4 .

But, from the recurrence relation, piq1 Ž x . s Ž xy a iq1 . pi Ž x . y biq1 piy1 Ž x . , that is, xpi Ž x . s piq1 Ž x . q a iq1 pi Ž x . q biq1 piy1 Ž x . . It follows that

Ha xp b

ny1

s

Ž x . pi Ž x . w Ž x . dx

Ha p b

ny1

Ž x . piq1 Ž x . q aiq1 pi Ž x . q biq1 piy1 Ž x . w Ž x . dx

s Ž pny1 ⭈ piq1 . ␻ q a iq1 Ž pny1 ⭈ pi . ␻ q biq1 Ž pny1 ⭈ piy1 . ␻ s 0. Hence, by Ž10.4., Ž pn ⭈ pi .␻ s 0.

I

It is easy to see from the recurrence relation Ž10.1. that pnŽ x . is of degree n, and the coefficient of x n is equal to one. Furthermore, we have the following corollaries: Corollary 10.1.1. An arbitrary polynomial of degree F n is uniquely expressible as a linear combination of p 0 Ž x ., p1Ž x ., . . . , pnŽ x .. Corollary 10.1.2.

The coefficient of x ny1 in pnŽ x . is yÝ nis1 a i Ž n G 1..

Proof. If d n denotes the coefficient of x ny1 in pnŽ x . Ž n G 2., then by comparing the coefficients of x ny1 on both sides of the recurrence relation

440

ORTHOGONAL POLYNOMIALS

Ž10.1., we obtain d n s d ny1 y a n ,

Ž 10.5 .

n s 2, 3, . . . .

The result follows from Ž10.5. and by noting that d1 s ya1

I

Another property of orthogonal polynomials is given by the following theorem: Theorem 10.1.2. If
pnŽ x .4⬁ns0 is a sequence of orthogonal polynomials with respect to w Ž x . on w a, b x, then the zeros of pnŽ x . Ž n G 1. are all real, distinct, and located in the interior of w a, b x. Proof. Since Ž pn ⭈ p 0 .␻ s 0 for n G 1, then Hab pnŽ x . w Ž x . dxs 0. This indicates that pnŽ x . must change sign at least once in Ž a, b . wrecall that w Ž x . is positivex. Suppose that pnŽ x . changes sign between a and b at just k points, denoted by x 1 , x 2 , . . . , x k . Let g Ž x . s Ž x y x 1 .Ž x y x 2 . ⭈⭈⭈ Ž x y x k .. Then, pnŽ x . g Ž x . is a polynomial with no zeros of odd multiplicity in Ž a, b .. Hence, Hab pnŽ x . g Ž x . w Ž x . dx/ 0, that is, Ž pn ⭈ g .␻ / 0. If k - n, then we have a contradiction by the fact that pn is orthogonal to g Ž x . w g Ž x ., being a polynomial of degree k, can be expressed as a linear combination of p 0 Ž x ., p1Ž x ., . . . , pk Ž x . by Corollary 10.1.1x. Consequently, k s n, and pnŽ x . has n distinct zeros in the interior of w a, b x. I Particular orthogonal polynomials can be derived depending on the choice of the interval w a, b x, and the weight function w Ž x .. For example, the well-known orthogonal polynomials listed below are obtained by the following selections of w a, b x and w Ž x .: a

Orthogonal Polynomial Legendre Jacobi Chebyshev of the first kind Chebyshev of the second kind Hermite Laguerre

b

y1 1 y1 1 y1 1 y1 1 y⬁ ⬁ 0 ⬁

wŽ x . 1 Ž1 y x . ␣ Ž1 q x . ␤ , ␣ , ␤ ) y1 Ž1 y x 2 .y1r2 Ž1 y x 2 .1r2 2 eyx r2 yx ␣ e x , ␣ ) y1

These polynomials are called classical orthogonal polynomials. We shall study their properties and methods of derivation. 10.2. LEGENDRE POLYNOMIALS These polynomials are derived by applying the so-called Rodrigues formula pn Ž x . s

1

d n Ž x 2 y 1.

2 n n!

dx n

n

,

n s 0, 1, 2, . . . .

441

LEGENDRE POLYNOMIALS

Thus, for n s 0, 1, 2, 3, 4, for example, we have p 0 Ž x . s 1, p1 Ž x . s x, p 2 Ž x . s 32 x 2 y 12 , p 3 Ž x . s 52 x 3 y 32 x, p4 Ž x . s 358 x 4 y 308 x 2 q 83 . From the Rodrigues formula it follows that pnŽ x . is a polynomial of degree n and the coefficient of x n is 2 n r2 n. We can multiply pnŽ x . by 2 nr 2 n to n n make the coefficient of x n equal to one Ž n s 1, 2, . . . .. Another definition of Legendre polynomials is obtained by means of the generating function,

ž /

ž /

g Ž x, r . s

1

Ž 1 y 2 rxq r 2 .

1r2

,

by expanding it as a power series in r for sufficiently small values of r. The coefficient of r n in this expansion is pnŽ x ., n s 0, 1, . . . , that is, g Ž x, r . s

⬁

Ý

pn Ž x . r n .

ns0

To demonstrate this, let us consider expanding Ž1 y z .y1r2 in a neighborhood of zero, where z s 2 xry r 2 : 1

Ž1yz.

1r2

s 1 q 12 z q 38 z 2 q 165 z 3 q ⭈⭈⭈ ,

z - 1, 2

3

s 1 q 12 Ž 2 xry r 2 . q 38 Ž 2 xry r 2 . q 165 Ž 2 xry r 2 . q ⭈⭈⭈ s 1 q xrq Ž 23 x 2 y 21 . r 2 q Ž 25 x 3 y 23 x . r 3 q ⭈⭈⭈ . We note that the coefficients of 1, r, r 2 , and r 3 are the same as p 0 Ž x ., p1Ž x ., p 2 Ž x ., p 3 Ž x ., as was seen earlier. In general, it is easy to see that the coefficient of r n is pnŽ x . Ž n s 0, 1, 2, . . . .. By differentiating g Ž x, r . with respect to r, it can be seen that

Ž 1 y 2 rxq r 2 .

⭸ g Ž x, r . ⭸r

y Ž x y r . g Ž x, r . s 0.

442

ORTHOGONAL POLYNOMIALS

By substituting g Ž x, r . s Ý⬁ns0 pnŽ x . r n in this equation, we obtain

Ž 1 y 2 rxq r 2 .

⬁

Ý

npn Ž x . r ny1 y Ž x y r .

ns1

⬁

Ý

pn Ž x . r n s 0.

ns0

The coefficient of r n must be zero for each n and for all values of x Ž n s 1, 2, . . . .. We thus have the following identity:

Ž n q 1 . pnq1 Ž x . y Ž 2 n q 1 . xpn Ž x . q npny1 Ž x . s 0,

n s 1, 2, . . . Ž 10.6 .

This is a recurrence relation that connects any three successive Legendre polynomials. For example, for p 2 Ž x . s 32 x 2 y 12 , p 3 Ž x . s 52 x 3 y 32 x, we find from Ž10.6. that p4 Ž x . s 14 7xp3 Ž x . y 3 p 2 Ž x . s 358 x 4 y 308 x 2 q 38 . 10.2.1. Expansion of a Function Using Legendre Polynomials 1 Suppose that f Ž x . is a function defined on wy1, 1x such that Hy1 f Ž x . pnŽ x . dx exists for n s 0, 1, 2, . . . . Consider the series expansion

⬁

f Ž x. s

Ý ai pi Ž x . .

Ž 10.7 .

is0

Multiplying both sides of Ž10.7. by pnŽ x . and then integrating from y1 to 1, we obtain, by the orthogonality of Legendre polynomials, an s

Hy1 p 1

2 n

y1

Ž x . dx

Hy1 f Ž x . p Ž x . dx, 1

n

n s 0, 1, . . . .

It can be shown that Žsee Jackson, 1941, page 52.

Hy1 p 1

2 n

Ž x . dxs

2 2nq1

,

n s 0, 1, 2, . . . .

Hence, the coefficient of pnŽ x . in Ž10.7. is given by an s

2nq1 2

Hy1 f Ž x . p Ž x . dx. 1

n

Ž 10.8 .

443

JACOBI POLYNOMIALS

If snŽ x . denotes the partial sum Ý nis0 a i pi Ž x . of the series in Ž10.7., then nq1

sn Ž x . s

2

f Ž t.

Hy1 t y x 1

pnq1 Ž t . pn Ž x . y pn Ž t . pnq1 Ž x . dt,

Ž 10.9 .

n s 0, 1, 2, . . . .

This is known as Christoffel’s identity Žsee Jackson, 1941, page 55.. If f Ž x . is continuous on wy1, 1x and has a derivative at xs x 0 , then lim n™⬁ snŽ x 0 . s f Ž x 0 ., and hence the series in Ž10.7. converges at x 0 to the value f Ž x 0 . Žsee Jackson, 1941, pages 64᎐65..

10.3. JACOBI POLYNOMIALS Jacobi polynomials, named after the German mathematician Karl Gustav JacobiŽ1804᎐1851., are orthogonal on wy1, 1x with respect to the weight function w Ž x . s Ž1 y x . ␣ Ž1 q x . ␤, ␣ ) y1, ␤ ) y1. The restrictions on ␣ and ␤ are needed to guarantee integrability of w Ž x . over the interval wy1, 1x. These polynomials, which we denote by pnŽ ␣ , ␤ . Ž x ., can be derived by applying the Rodrigues formula: pnŽ ␣ , ␤ .

Ž x. s

Ž y1.

n

2 n n!

Ž1yx.

y␣

Ž1qx.

y␤

dn Ž1yx .

␣qn

Ž1qx.

␤qn

dx n n s 0, 1, 2, . . . .

,

Ž 10.10 .

This formula reduces to the one for Legendre polynomials when ␣ s ␤ s 0. Thus, Legendre polynomials represent a special class of Jacobi polynomials. Applying the so-called Leibniz formula Žsee Exercise 4.2 in Chapter 4. concerning the nth derivative of a product of two functions, namely, f nŽ x . s Ž1 y x . ␣qn and g nŽ x . s Ž1 q x . ␤qn in Ž10.10., we obtain dn Ž1yx .

␣qn

Ž1qx.

dx n

␤qn

n

s

Ý

is0

ž/

n Ž i. f Ž x . g nŽ nyi . Ž x . , i n

n s 0, 1, . . . ,

Ž 10.11 . where for i s 0, 1, . . . , n, f nŽ i. Ž x . is a constant multiple of Ž1 y x . ␣qnyi s Ž1 y x . ␣ Ž1 y x . nyi , and g nŽ nyi . Ž x . is a constant multiple of Ž1 q x . ␤qi s Ž1 q x . ␤ Ž1 q x . i. Thus, the nth derivative in Ž10.11. has Ž1 y x . ␣ Ž1 q x . ␤ as a factor. Using formula Ž10.10., it can be shown that pnŽ ␣ , ␤ . Ž x . is a polynomial of degree n with the leading coefficient Žthat is, the coefficient of x n . equal to Ž1r2 n n!. ⌫ Ž2 n q ␣ q ␤ q 1.r⌫ Ž n q ␣ q ␤ q 1..

444

ORTHOGONAL POLYNOMIALS

10.4. CHEBYSHEV POLYNOMIALS These polynomials were named after the Russian mathematician Pafnuty Lvovich Chebyshev Ž1821᎐1894.. In this section, two kinds of Chebyshev polynomials will be studied, called, Chebyshev polynomials of the first kind and of the second kind. 10.4.1. Chebyshev Polynomials of the First Kind These polynomials are denoted by TnŽ x . and defined as Tn Ž x . s cos Ž n Arccos x . ,

n s 0, 1, . . . ,

Ž 10.12 .

where 0 F Arccos x F ␲ . Note that TnŽ x . can be expressed as Tn Ž x . s x n q

ž/

ž/

2 n ny2 2 n ny4 2 x x Ž x y 1. q Ž x y 1 . q ⭈⭈⭈ , 2 4

n s 0, 1, . . . ,

Ž 10.13 .

where y1 F xF 1. Historically, the polynomials defined by Ž10.13. were originally called Chebyshev polynomials without any qualifying expression. Using Ž10.13., it is easy to obtain the first few of these polynomials: T0 Ž x . s 1, T1 Ž x . s x, T2 Ž x . s 2 x 2 y 1, T3 Ž x . s 4 x 3 y 3 x, T4 Ž x . s 8 x 4 y 8 x 2 q 1, T5 Ž x . s 16 x 5 y 20 x 3 q 5 x, . . . The following are some properties of TnŽ x .: 1. y1 F TnŽ x . F 1 for y1 F xF 1. 2. TnŽyx . s Žy1. n TnŽ x .. 3. TnŽ x . has simple zeros at the following n points:

␰ i s cos

Ž 2 i y 1. ␲ 2n

,

i s 1, 2, . . . , n.

445

CHEBYSHEV POLYNOMIALS

We may recall that these zeros, also referred to as Chebyshev points, were instrumental in minimizing the error of Lagrange interpolation in Chapter 9 Žsee Theorem 9.2.3.. 4. The weight function for TnŽ x . is w Ž x . s Ž1 y x 2 .y1r2 . To show this, we have that for two nonnegative integers, m, n, ␲

H0 cos m␪ cos n␪ d␪ s 0,

m / n,

Ž 10.14 .

and ␲

H0 cos

2

n ␪ d␪ s

½

␲r2, ␲,

n / 0, . ns0

Ž 10.15 .

Making the change of variables xs cos ␪ in Ž10.14. and Ž10.15., we obtain

Hy1 1

Tm Ž x . Tn Ž x .

Hy1 1

Ž1yx2 .

1r2

Tn2 Ž x . 2 1r2

Ž1yx .

dxs 0,

dxs

½

m / n,

␲r2, ␲,

n / 0, n s 0.

This shows that
TnŽ x .4⬁ns0 forms a sequence of orthogonal polynomials on wy1, 1x with respect to w Ž x . s Ž1 y x 2 .y1r2 . 5. We have Tnq1 Ž x . s 2 xTn Ž x . y Tny1 Ž x . ,

n s 1, 2, . . . .

Ž 10.16 .

To show this recurrence relation, we use the following trigonometric identities: cos Ž n q 1 . ␪ s cos n␪ cos ␪ y sin n␪ sin ␪ , cos Ž n y 1 . ␪ s cos n␪ cos ␪ q sin n␪ sin ␪ . Adding these identities, we obtain cos Ž n q 1 . ␪ s 2 cos n␪ cos ␪ y cos Ž n y 1 . ␪ .

Ž 10.17 .

If we set xs cos ␪ and cos n␪ s TnŽ x . in Ž10.17., we obtain Ž10.16.. Recall that T0 Ž x . s 1 and T1Ž x . s x. 10.4.2. Chebyshev Polynomials of the Second Kind These polynomials are defined in terms of Chebyshev polynomials of the first kind as follows: Differentiating TnŽ x . s cos n␪ with respect to xs cos ␪ , we

446

ORTHOGONAL POLYNOMIALS

obtain, dTn Ž x . dx

s yn sin n␪ sn

sin n␪ sin ␪

d␪ dx

.

Let UnŽ x . be defined as Un Ž x . s s

1

dTnq1 Ž x .

nq1

dx

sin Ž n q 1 . ␪ sin ␪

,

n s 0, 1, . . .

Ž 10.18 .

This polynomial, which is of degree n, is called a Chebyshev polynomial of the second kind. Note that Un Ž x . s

sin n␪ cos ␪ q cos n␪ sin ␪ sin ␪

s xUny1 Ž x . q Tn Ž x . ,

n s 1, 2, . . . ,

Ž 10.19 .

where U0 Ž x . s 1. Formula Ž10.19. provides a recurrence relation for UnŽ x .. Another recurrence relation that is free of TnŽ x . can be obtained from the following identity: sin Ž n q 1 . ␪ s 2 sin n␪ cos ␪ y sin Ž n y 1 . ␪ . Hence, Un Ž x . s 2 xUny1 Ž x . y Uny2 Ž x . ,

n s 2, 3, . . . .

Ž 10.20 .

Using the fact that U0 Ž x . s 1, U1Ž x . s 2 x, formula Ž10.20. can be used to derive expressions for UnŽ x ., n s 2, 3, . . . . It is easy to see that the leading coefficient of x n in UnŽ x . is 2 n. We now show that
UnŽ x .4⬁ns0 forms a sequence of orthogonal polynomials with respect to the weight function, w Ž x . s Ž1 y x 2 .1r2 , over wy1, 1x. From the formula ␲

H0 sin

Ž m q 1 . ␪ sin Ž n q 1 . ␪ d␪ s 0,

m / n,

447

HERMITE POLYNOMIALS

we get, after making the change of variables x s cos ␪ ,

Hy1U 1

m

Ž x . Un Ž x . Ž 1 y x 2 .

1r2

m / n.

dxs 0,

This shows that w Ž x . s Ž1 y x 2 .1r2 is a weight function for the sequence 1
UnŽ x .4⬁ns0 . Note that Hy1 Un2 Ž x .Ž1 y x 2 .1r2 dxs ␲r2. 10.5. HERMITE POLYNOMIALS Hermite polynomials, denoted by
HnŽ x .4⬁ns0 , were named after the French mathematician Charles Hermite Ž1822᎐1901.. They are defined by the Rodrigues formula, n

Hn Ž x . s Ž y1 . e

x 2 r2

d n Ž eyx dx

2

r2

.

n

n s 0, 1, 2, . . . .

,

Ž 10.21 .

From Ž10.21., we have d n Ž eyx dx

2

r2

n

.

n

s Ž y1 . eyx

2

r2

Hn Ž x . .

Ž 10.22 .

By differentiating the two sides in Ž10.22., we obtain d nq1 Ž eyx dx

2

r2

nq1

.

s Ž y1 .

n

yxeyx

2

r2

Hn Ž x . q eyx

2

r2

dHn Ž x . dx

. Ž 10.23 .

But, from Ž10.21., d nq1 Ž eyx dx

nq1

2

r2

.

s Ž y1 .

nq1 yx 2 r2

e

Hnq1 Ž x . .

Ž 10.24 .

From Ž10.23. and Ž10.24. we then have Hnq1 Ž x . s xHn Ž x . y

dHn Ž x . dx

,

n s 0, 1, 2, . . . ,

which defines a recurrence relation for the sequence
HnŽ x .4⬁ns0 . Since H0 Ž x . s 1, it follows by induction, using this relation, that HnŽ x . is a polynomial of degree n. Its leading coefficient is equal to one.

448

ORTHOGONAL POLYNOMIALS

Note that if w Ž x . s eyx

2

r2

, then

ž

w Ž xy t . s exp y

x2

q txy

2

ž

s w Ž x . exp txy

t2 2

t2 2

/

/

.

Applying Taylor’s expansion to w Ž x y t ., we obtain w Ž xy t . s

⬁

Ý

Ž y1. n!

ns0

s

⬁

tn

ns0

n!

Ý

n

t

n

dn wŽ x . dx n

Hn Ž x . w Ž x . .

Consequently, HnŽ x . is the coefficient of t nrn! in the expansion of expŽ txy t 2r2.. It follows that Hn Ž x . s x n y

nw2x 2.1!

x ny2 q

nw4x 2 2 ⭈ 2!

x ny4 y

nw6x 2 3 ⭈ 3!

x ny6 q ⭈⭈⭈ ,

where nw r x s nŽ n y 1.Ž n y 2. ⭈⭈⭈ Ž n y r q 1.. This particular representation of HnŽ x . is given in Kendall and Stuart Ž1977, page 167.. For example, the first seven Hermite polynomials are H0 Ž x . s 1, H1 Ž x . s x, H2 Ž x . s x 2 y 1, H3 Ž x . s x 3 y 3 x, H4 Ž x . s x 4 y 6 x 2 q 3, H5 Ž x . s x 5 y 10 x 3 q 15 x, H6 Ž x . s x 6 y 15 x 4 q 45 x 2 y 15, . . . Another recurrence relation that does not use the derivative of HnŽ x . is given by Hnq1 Ž x . s xHn Ž x . y nHny1 Ž x . ,

n s 1, 2, . . . ,

Ž 10.25 .

449

HERMITE POLYNOMIALS

with H0 Ž x . s 1 and H1Ž x . s x. To show this, we use Ž10.21. in Ž10.25.:

Ž y1.

nq1

e

x 2 r2

d nq1 Ž eyx dx n

s x Ž y1 . e x

2

r2

2

.

r2

nq1

d n Ž eyx dx

2

r2

.

y n Ž y1 .

n

ny1

ex

2

r2

d ny1 Ž eyx dx

2

r2

.

ny1

or equivalently, y

d nq1 Ž eyx dx

2

r2

.

d n Ž eyx

sx

nq1

dx

2

.

r2

n

qn

d ny1 Ž eyx dx

2

r2

.

ny1

.

This is true given the fact that d Ž eyx

2

r2

.

s yxeyx

dx

2

r2

.

Hence, y

d nq1 Ž eyx

2

r2

.

s

dx nq1

d n Ž xeyx

2

r2

.

dx n

sn

d ny1 Ž eyx dx

2

r2

.

qx

ny1

d n Ž eyx dx

2

r2

.

n

Ž 10.26 .

,

which results from applying Leibniz’s formula to the right-hand side of Ž10.26.. We now show that
HnŽ x .4⬁ns0 forms a sequence of orthogonal polynomials 2 with respect to the weight function w Ž x . s eyx r2 over Žy⬁, ⬁.. For this purpose, let m, n be nonnegative integers, and let c be defined as cs

⬁

Hy⬁e

yx 2 r2

Hm Ž x . Hn Ž x . dx.

Ž 10.27 .

Then, from Ž10.21. and Ž10.27., we have c s Ž y1 .

n

⬁

Hy⬁ H

d n Ž eyx

mŽ x .

dx

2

r2

.

n

dx.

Integrating by parts gives c s Ž y1 .

s Ž y1 .

n

½

nq1

Hm Ž x . ⬁

Hy⬁

d ny1 Ž eyx dx

2

r2

.

⬁

y

ny1 y⬁

dHm Ž x . d ny1 Ž eyx dx

dx

ny1

2

r2

.

⬁

Hy⬁ dx.

dHm Ž x . d ny1 Ž eyx dx

dx

ny1

2

r2

.

dx

5

Ž 10.28 .

450

ORTHOGONAL POLYNOMIALS

Formula Ž10.28. is true because HmŽ x . d ny1 Ž eyx r2 .rdx ny1 , which is a poly2 nomial multiplied by eyx r2 , has a limit equal to zero as x™ .⬁. By repeating the process of integration by parts m y 1 more times, we obtain for n)m 2

c s Ž y1 .

d nym Ž eyx

d m Hm Ž x .

⬁

Hy⬁

mqn

dx

m

dx

2

r2

.

nym

dx.

Ž 10.29 .

Note that since HmŽ x . is a polynomial of degree m with a leading coefficient equal to one, d m w HmŽ x .xrdx m is a constant equal to m!. Furthermore, since n ) m, then ⬁

Hy⬁

d nym Ž eyx dx

2

r2

.

nym

dxs 0.

It follows that c s 0. We can also arrive at the same conclusion if n - m. Hence, ⬁

Hy⬁e

yx 2 r2

Hm Ž x . Hn Ž x . dxs 0,

m / n.

This shows that
HnŽ x .4⬁ns0 is a sequence of orthogonal polynomials with 2 respect to w Ž x . s eyx r2 over Žy⬁, ⬁.. Note that if m s n in Ž10.29., then cs

⬁

Hy⬁

s n!

d n Hn Ž x . dx n ⬁

Hy⬁e

s 2 n!

yx 2 r2

⬁

H0 e

yx 2 r2

eyx

2

r2

dx

dx dx

s n!'2␲ By comparison with Ž10.27., we conclude that ⬁

Hy⬁e

yx 2 r2

Hn2 Ž x . dxs n!'2␲ .

Ž 10.30 .

Hermite polynomials can be used to provide the following series expansion of a function f Ž x .: f Ž x. s

⬁

Ý c n Hn Ž x . ,

ns0

Ž 10.31 .

451

LEGUERRE POLYNOMIALS

where cn s

1

⬁

H e n!'2␲ y⬁

yx 2 r2

f Ž x . Hn Ž x . dx,

n s 0, 1, . . . .

Ž 10.32 .

Formula Ž10.32. follows from multiplying both sides of Ž10.31. with 2 eyx r2 HnŽ x ., integrating over Žy⬁, ⬁., and noting formula Ž10.30. and the orthogonality of the sequence
HnŽ x .4⬁ns0 . 10.6. LAGUERRE POLYNOMIALS Laguerre polynomials were named after the French mathematician Edmond Laguerre Ž1834᎐1886.. They are denoted by LŽn␣ . Ž x . and are defined over the interval Ž0, ⬁., n s 0, 1, 2, . . . , where ␣ ) y1. The development of these polynomials is based on an application of Leibniz formula to finding the nth derivative of the function

␾n Ž x . s x ␣qn eyx . More specifically, for ␣ ) y1, LŽn␣ . Ž x . is defined by a Rodrigues-type formula, namely, n

LŽn␣ . Ž x . s Ž y1 . e x xy␣

d n Ž x nq␣ eyx . dx n

,

n s 0, 1,2, . . . .

We shall henceforth use L nŽ x . instead of LŽn␣ . Ž x .. From this definition, we conclude that L nŽ x . is a polynomial of degree n with a leading coefficient equal to one. It can also be shown that Laguerre polynomials are orthogonal with respect to the weight function w Ž x . s eyx x ␣ over Ž0, ⬁., that is, ⬁

H0 e

yx

x ␣ L m Ž x . L n Ž x . dxs 0,

m/n

Žsee Jackson, 1941, page 185.. Furthermore, if m s n, then ⬁

H0 e

yx

x ␣ Ln Ž x .

2

dxs n!⌫ Ž ␣ q n q 1 . ,

n s 0, 1, . . . .

A function f Ž x . can be expressed as an infinite series of Laguerre polynomials of the form f Ž x. s

⬁

Ý cn Ln Ž x . ,

ns0

452

ORTHOGONAL POLYNOMIALS

where cn s

1

⬁

He n!⌫ Ž ␣ q n q 1 . 0

yx

x ␣ L n Ž x . f Ž x . dx,

n s 0, 1, 2, . . . .

A recurrence relation for L nŽ x . is developed as follows: From the definition of L nŽ x ., we have n

Ž y1. x ␣ eyx L n Ž x . s

d n Ž x nq␣ eyx . dx n

Ž 10.33 .

n s 0, 1, 2, . . . .

,

Replacing n by n q 1 in Ž10.33. gives

Ž y1.

nq1

x ␣ eyx L nq1 Ž x . s

d nq1 Ž x nq␣q1 eyx . dx nq1

Ž 10.34 .

.

Now, x nq␣q1 eyx s x Ž x nq␣ eyx . .

Ž 10.35 .

Applying the Leibniz formula for the Ž n q 1.st derivative of the product on the right-hand side of Ž10.35. and noting that the nth derivative of x is zero for n s 2, 3, 4, . . . , we obtain d nq1 Ž x nq␣q1 eyx . dx nq1

sx sx

d nq1 Ž x nq␣ eyx . dx nq1

q Ž n q 1.

d

d n Ž x nq␣ eyx .

dx

dx n

d n Ž x nq␣ eyx . dx n

q Ž n q 1.

d n Ž x nq␣ eyx . dx n

, Ž 10.36 .

Using Ž10.33. and Ž10.34. in Ž10.36. gives

Ž y1.

nq1

sx

x ␣ eyx L nq1 Ž x . d

dx

n

n

Ž y1. x ␣ eyx L n Ž x . q Ž y1. Ž n q 1 . x ␣ eyx L n Ž x . n

s Ž y1 . x ␣ eyx Ž ␣ q n q 1 y x . L n Ž x . q x

dL n Ž x . dx

.

Ž 10.37 .

Multiplying the two sides of Ž10.37. by Žy1. nq1 e x xy␣ , we obtain L nq1 Ž x . s Ž xy ␣ y n y 1 . L n Ž x . y x

dLn Ž x . dx

,

n s 0, 1, 2, . . . .

This recurrence relation gives L nq1 Ž x . in terms of L nŽ x . and its derivative. Note that L0 Ž x . s 1.

LEAST-SQUARES APPROXIMATION WITH ORTHOGONAL POLYNOMIALS

453

Another recurrence relation that does not require using the derivative of L nŽ x . is given by Žsee Jackson, 1941, page 186. L nq1 Ž x . s Ž xy ␣ y 2 n y 1 . L n Ž x . y n Ž ␣ q n . L ny1 Ž x . ,

n s 1, 2, . . . .

10.7. LEAST-SQUARES APPROXIMATION WITH ORTHOGONAL POLYNOMIALS In this section, we consider an approximation problem concerning a continun ous function. Suppose that we have a set of polynomials,
pi Ž x .4is0 , orthogonal with respect to a weight function w Ž x . over the interval w a, b x. Let f Ž x . be a continuous function on w a, b x. We wish to approximate f Ž x . by the sum Ý nis0 c i pi Ž x ., where c 0 , c1 , . . . , c n are constants to be determined by minimizing the function

␥ Ž c 0 , c1 , . . . , c n . s

n

Ha Ý c p Ž x . y f Ž x . b

i

i

2

w Ž x . dx,

is0

that is, ␥ is the square of the norm, 5Ý nis0 c i pi y f 5 ␻ . If we differentiate ␥ with respect to c 0 , c1 , . . . , c n and equate the partial derivatives to zero, we obtain

⭸␥ ⭸ ci

n

Ha Ý c p Ž x . y f Ž x .

s2

b

j

j

pi Ž x . w Ž x . dxs 0,

i s 0, 1, . . . , n.

js0

Hence, n

Ý

js0

Ha p Ž x . p Ž x . w Ž x . dx b

i

cj s

j

Ha f Ž x . p Ž x . w Ž x . dx, b

i

i s 0, 1, . . . , n.

Ž 10.38 . Equations Ž10.38. can be written in vector form as

Ž 10.39 .

Sc s u,

where c s Ž c 0 , c1 , . . . , c n .X , u s Ž u 0 , u1 , . . . , u n .X with u i s Hab f Ž x . pi Ž x . w Ž x . dx, and S is an Ž n q 1. = Ž n q 1. matrix whose Ž i, j .th element, si j , is given by si j s

Ha p Ž x . p Ž x . w Ž x . dx, b

i

i , j s 0, 1, . . . , n.

j

Since p 0 Ž x ., p1Ž x ., . . . , pnŽ x . are orthogonal, then S must be a diagonal matrix with diagonal elements given by si i s

Ha p

b 2 i

Ž x . w Ž x . dxs pi

2 ␻,

i s 0, 1, . . . , n.

454

ORTHOGONAL POLYNOMIALS

From equation Ž10.39. we get the solution c s Sy1 u. The ith element of c is therefore of the form ci s s

ui sii

Ž f ⭈ pi . ␻ pi

2 ␻

i s 0, 1, . . . , n.

,

For such a value of c, ␥ has an absolute minimum, since S is positive definite. It follows that the linear combination pUn Ž x . s

n

Ý c i pi Ž x .

is0

s

n

Ž f ⭈ pi . ␻

is0

pi

Ý

2 ␻

pi Ž x .

Ž 10.40 .

minimizes ␥ . We refer to pUn Ž x . as the least-squares polynomial approximation of f Ž x . with respect to p 0 Ž x ., p1Ž x ., . . . , pnŽ x .. If
pnŽ x .4⬁ns0 is a sequence of orthogonal polynomials, then pUn Ž x . in Ž10.40. represents a partial sum of the infinite series Ý⬁ns0 wŽ f ⭈ pn .␻ r 5 pn 5 2␻ x pnŽ x .. This series may fail to converge point by point to f Ž x .. It converges, however, to f Ž x . in the norm 5 ⭈ 5 ␻ . This is shown in the next theorem. Theorem 10.7.1.

If f : w a, b x ™ R is continuous, then

Ha

b

f Ž x . y pUn Ž x . w Ž x . dx™ 0 2

as n ™ ⬁, where pUn Ž x . is defined by formula Ž10.40.. Proof. By the Weierstrass theorem ŽTheorem 9.1.1., there exists a polynomial bnŽ x . of degree n that converges uniformly to f Ž x . on w a, b x, that is, f Ž x . y bn Ž x . ™ 0

sup

as n ™ ⬁.

aFxFb

Hence,

Ha

b

2

f Ž x . y bn Ž x . w Ž x . dx™ 0

as n ™ ⬁, since

Ha

b

2

f Ž x . y bn Ž x . w Ž x . dxF sup aFxFb

f Ž x . y bn Ž x .

2

Ha w Ž x . dx. b

455

ORTHOGONAL POLYNOMIALS DEFINED ON A FINITE SET

Furthermore,

Ha

b

f Ž x . y pUn Ž x . w Ž x . dxF 2

Ha

b

2

f Ž x . y bn Ž x . w Ž x . dx, Ž 10.41 .

since, by definition, pUn Ž x . is the least-squares polynomial approximation of f Ž x .. From inequality Ž10.41. we conclude that 5 f y pUn 5 ␻ ™ 0 as n ™ ⬁. I 10.8. ORTHOGONAL POLYNOMIALS DEFINED ON A FINITE SET In this section, we consider polynomials, p 0 Ž x ., p1Ž x ., . . . , pnŽ x ., defined on a finite set D s
x 0 , x 2 , . . . , x n4 such that aF x i F b, i s 0, 1, . . . , n. These polynomials are orthogonal with respect to a weight function wU Ž x . over D if n

Ý wU Ž x i . pm Ž x i . p␯ Ž x i . s 0,

m/␯ ;

m, ␯ s 0, 1, . . . , n.

is0

Such polynomials are said to be orthogonal of the discrete type. For example, n the set of discrete Chebyshev polynomials,
t i Ž j, n.4is0 , which are defined over the set of integers j s 0, 1, . . . , n, are orthogonal with respect to wU Ž j . s 1, j s 0, 1, 2, . . . , n, and are given by the following formula Žsee Abramowitz and Stegun, 1964, page 791.:

ž /ž / Ž Ž

i

t i Ž j, n . s

iqk k

i k

Ý Ž y1. k

ks0

i s 0, 1, . . . , n;

j! n y k . ! j y k . !n!

,

j s 0, 1, . . . , n.

For example, for i s 0, 1, 2, we have t 0 Ž j, n . s 1, t 1 Ž j, n . s 1 y t 2 Ž j, n . s 1 y s1y s1y

j s 0, 1, 2, . . . , n, 2 n 6 n 6j n 6j n

j s 0, 1, . . . , n,

j, jy

ž ž

j Ž j y 1.

1y

ny1 jy1 ny1

nyj ny1

/

,

/ j s 0, 1, 2, . . . , n.

Ž 10.42 .

456

ORTHOGONAL POLYNOMIALS

A recurrence relation for the discrete Chebyshev polynomials is of the form

Ž i q 1 . Ž n y i . t iq1 Ž j, n . s Ž 2 i q 1. Ž n y 2 j . t i Ž j, n . y i Ž n q i q 1. t iy1 Ž j, n . , i s 1, 2, . . . , n.

Ž 10.43 .

10.9. APPLICATIONS IN STATISTICS Orthogonal polynomials play an important role in approximating distribution functions of certain random variables. In this section, we consider only univariate distributions. 10.9.1. Applications of Hermite Polynomials Hermite polynomials provide a convenient tool for approximating density functions and quantiles of distributions using convergent series. They are associated with the normal distribution, and it is therefore not surprising that they come up in various investigations in statistics and probability theory. Here are some examples. 10.9.1.1. Approximation of Density Functions and Quantiles of Distributions Let ␾ Ž x . denote the density function of the standard normal distribution, that is,

␾Ž x. s

1

'2␲

eyx

2

r2

y⬁ - x- ⬁.

,

Ž 10.44 .

We recall from Section 10.5 that the sequence
HnŽ x .4⬁ns0 of Hermite 2 polynomials is orthogonal with respect to w Ž x . s eyx r2 , and that Hn Ž x . s x n y

nw2x 2 ⭈ 1!

x ny2 q

nw4x 2 2 ⭈ 2!

x ny4 y

nw6x 2 3 ⭈ 3!

x ny6 q ⭈⭈⭈ , Ž 10.45 .

where nw r x s nŽ n y 1.Ž n y 2. ⭈⭈⭈ Ž n y r q 1.. Suppose now that g Ž x . is a density function for some continuous distribution. We can represent g Ž x . as a series of the form gŽ x. s

⬁

Ý bn Hn Ž x . ␾ Ž x . ,

Ž 10.46 .

ns0

where, as in formula Ž10.32., bn s

1

⬁

H g Ž x . H Ž x . dx. n! y⬁ n

Ž 10.47 .

457

APPLICATIONS IN STATISTICS

By substituting HnŽ x ., as given by formula Ž10.45., in formula Ž10.47., we obtain an expression for bn in terms of the central moments, ␮ 0 , ␮ 1 , . . . , ␮ n , . . . , of the distribution whose density function is g Ž x .. These moments are defined as

␮n s

⬁

Hy⬁ Ž xy ␮ .

n

g Ž x . dx,

n s 0, 1, 2, . . . ,

where ␮ is the mean of the distribution. Note that ␮ 0 s 1, ␮ 1 s 0, and ␮ 2 s ␴ 2 , the variance of the distribution. In particular, if ␮ s 0, then b 0 s 1, b1 s 0, b 2 s 12 Ž ␮ 2 y 1 . , b 3 s 16 ␮ 3 , b4 s 241 Ž ␮4 y 6 ␮ 2 q 3 . , 1 b5 s 120 Ž ␮5 y 10 ␮ 3 . , 1 b6 s 720 Ž ␮6 y 15␮4 q 45␮ 2 y 15. , . . .

The expression for g Ž x . in formula Ž10.46. can then be written as g Ž x . s ␾ Ž x . 1 q 12 Ž ␮ 2 y 1 . H2 Ž x . q 16 ␮ 3 H3 Ž x . q 241 Ž ␮4 y 6 ␮ 2 q 3 . H4 Ž x . q ⭈⭈⭈ .

Ž 10.48 .

This expression is known as the Gram᎐Charlier series of type A. Thus the Gram᎐Charlier series provides an expansion of g Ž x . in terms of its central moments, the standard normal density, and Hermite polynomials. Using formulas Ž10.21. and Ž10.46., we note that g Ž x . can be expressed as a series of derivatives of ␾ Ž x . of the form gŽ x. s

⬁

Ý

ns0

cn n!

ž / d

dx

n

␾Ž x. ,

where c n s Ž y1 .

n

⬁

Hy⬁ g Ž x . H Ž x . dx, n

n s 0, 1, . . . .

Ž 10.49 .

458

ORTHOGONAL POLYNOMIALS

Cramer ´ Ž1946, page 223. gave conditions for the convergence of the series on the right-hand side of formula Ž10.49., namely, if g Ž x . is continuous and ⬁ of bounded variation on Žy⬁, ⬁., and if the integral Hy⬁ g Ž x . expŽ x 2r4. dx is convergent, then the series in formula Ž10.49. will converge for every x to g Ž x .. We can utilize Gram᎐Charlier series to find the upper ␣-quantile, x␣ , of the distribution with the density function g Ž x .. This point is defined as

Hy⬁ g Ž x . dxs 1 y ␣ . x␣

From Ž10.46. we have that g Ž x. s␾ Ž x. q

⬁

Ý bn Hn Ž x . ␾ Ž x . .

ns2

Then ⬁

Hy⬁ g Ž x . dxs Hy⬁ ␾ Ž x . dxq Ý b Hy⬁ H Ž x . ␾ Ž x . dx. x␣

x␣

x␣

n

n

Ž 10.50 .

ns2

However,

Hy⬁ H Ž x . ␾ Ž x . dxs yH x␣

n

ny1

Ž x␣ . ␾ Ž x␣ . .

To prove this equality we note that by formula Ž10.21.

Hy⬁ x␣

ž / Ž. . ž / Ž .

Hn Ž x . ␾ Ž x . dxs Ž y1 . s Ž y1

n

Hy⬁ x␣

␾ x dx

dx

ny1

d

n

n

d

␾ x␣ ,

dx

where Ž drdx . ny1␾ Ž x␣ . denotes the value of the Ž n y 1.st derivative of ␾ Ž x . at x␣ . By applying formula Ž10.21. again we obtain

Hy⬁ H Ž x . ␾ Ž x . dxs Ž y1. x␣

n

n

Ž y1.

ny1

Hny1 Ž x␣ . ␾ Ž x␣ .

s yHny1 Ž x␣ . ␾ Ž x␣ . . By making the substitution in formula Ž10.50., we get ⬁

Hy⬁ g Ž x . dxs Hy⬁ ␾ Ž x . dxy Ý b H x␣

x␣

n

ns2

ny1

Ž x␣ . ␾ Ž x␣ . .

Ž 10.51 .

459

APPLICATIONS IN STATISTICS

Now, suppose that z␣ is the upper ␣-quantile of the standard normal distribution. Then

Hy⬁ g Ž x . dxs 1 y ␣ s Hy⬁ ␾ Ž x . dx. x␣

z␣

Using the expansion Ž10.51., we obtain ⬁

Hy⬁␾ Ž x . dxy Ý b H x␣

n

Ž x␣ . ␾ Ž x␣ . s H ␾ Ž x . dx. z␣

ny1

ns2

y⬁

Ž 10.52 .

If we expand the right-hand side of equation Ž10.52. using Taylor’s series in a neighborhood of x␣ , we get

Ž z␣ y x ␣ .

⬁

Hy⬁␾ Ž x . dxs Hy⬁ ␾ Ž x . dxq Ý z␣

x␣

j!

js1

s

Hy⬁␾ Ž x . dxq Ý

jy1

␾ Ž x␣ .

j

Ž y1.

j!

js1

ž / d

dx

Ž z␣ y x ␣ .

⬁

x␣

j

jy1

H jy1 Ž x␣ . ␾ Ž x␣ . ,

using formula Ž 10.21 . s

Hy⬁ x␣

␾ Ž x . dxy

Ž x ␣ y z␣ .

⬁

Ý

j!

js1

j

H jy1 Ž x␣ . ␾ Ž x␣ . .

Ž 10.53 .

From formulas Ž10.52. and Ž10.53. we conclude that ⬁

Ý

bn Hny1 Ž x␣ . ␾ Ž x␣ . s

ns2

⬁

Ý

Ž x ␣ y z␣ .

j

H jy1 Ž x␣ . ␾ Ž x␣ . .

j!

js1

By dividing both sides by ␾ Ž x␣ . we obtain ⬁

⬁

ns2

js1

Ý bn Hny1 Ž x␣ . s Ý

Ž x ␣ y z␣ . j!

j

H jy1 Ž x␣ . .

Ž 10.54 .

This provides a relationship between x␣ , the ␣-quantile of the distribution with the density function g Ž x ., and z␣ , the corresponding quantile for the standard normal. Since the bn’s are functions of the moments associated with g Ž x ., then it is possible to use Ž10.54. to express x␣ in terms of z␣ and the moments of g Ž x .. This was carried out by Cornish and Fisher Ž1937.. They provided an expansion for x␣ in terms of z␣ and the cumulants Žinstead of the moments. associated with g Ž x .. ŽSee Section 5.6.2 for a definition of cumulants. Note that there is a one-to-one correspondence between moments and cumulants.. Such an expansion became known as the

460

ORTHOGONAL POLYNOMIALS

Cornish᎐Fisher expansion. It is reported in Johnson and Kotz Ž1970, page 34. Žsee Exercise 10.11.. See also Kendall and Stuart Ž1977, pages 175᎐178.. 10.9.1.2. Approximation of a Normal Integral A convergent series representing the integral 1

'2␲ Ho e

␺ Ž x. s

x

yt 2 r2

dt

was derived by Kerridge and Cook Ž1976.. Their method is based on the fact that

ž /

2 nq1 x Ž xr2. f Ž2 n. 2 Ž 2 n q 1. !

⬁

H0 f Ž t . dts 2 Ý x

ns0

Ž 10.55 .

for any function f Ž t . with a suitably convergent Taylor’s expansion in a neighborhood of xr2, namely, f Ž t. s

⬁

1

ns0

n!

Ý

ž

ty

x 2

/ ž / n

f Ž n.

x

2

Ž 10.56 .

.

Formula Ž10.55. results from integrating Ž10.56. with respect to t from 0 to x 2 and noting that the even terms vanish. Taking f Ž t . s eyt r2 , we obtain

H0

x

eyt

2

r2

2 nq1 d 2 n Ž eyt Ž xr2. dt 2 n Ž 2 n q 1. !

2

⬁

dts2

Ý

ns0

r2

.

.

Ž 10.57 .

tsxr2

Using the Rodrigues formula for Hermite polynomials wformula Ž10.21.x, we get d n Ž eyx

2

r2

.

n

dx

n

s Ž y1 . eyx

2

r2

Hn Ž x . ,

n s 0, 1, . . . .

By making the substitution in Ž10.57., we find

H0 e x

yt 2 r2

dts 2

Ý

ns0

s 2 eyx

ž / ž /

2 nq1 x Ž xr2. 2 eyx r2 H2 n 2 Ž 2 n q 1. !

⬁

2

r8

⬁

Ž xr2.

2 nq1

Ý ns0 Ž 2 n q 1 . !

x

H2 n

2

.

Ž 10.58 .

This expression can be simplified by letting ⌰nŽ x . s x n HnŽ x .rn! in Ž10.58., which gives

H0 e x

yt 2 r2

dts xeyx

2

r8

⬁

Ý

ns0

⌰ 2 n Ž xr2. 2nq1

.

461

APPLICATIONS IN STATISTICS

Hence,

␺ Ž x. s

1

'2␲

xeyx

2

r8

⬁

Ý

⌰ 2 n Ž xr2.

ns0

2nq1

.

Ž 10.59 .

Note that on the basis of formula Ž10.25., the recurrence relation for ⌰nŽ x . is given by ⌰nq1 s

x 2 Ž ⌰n y ⌰ny1 . nq1

,

n s 1, 2, . . . .

The ⌰nŽ x .’s are easier to handle numerically than the Hermite polynomials, as they remain relatively small, even for large n. Kerridge and Cook Ž1976. report that the series in Ž10.59. is accurate over a wide range of x. Divgi Ž1979., however, states that the convergence of the series becomes slower as x increases. 10.9.1.3. Estimation of Unknown Densities Let X 1 , X 2 , . . . , X n represent a sequence of independent random variables with a common, but unknown, density function f Ž x . assumed to be square integrable. From Ž10.31. we have the representation f Ž x. s

⬁

Ý c j Hj Ž x . ,

js0

or equivalently, f Ž x. s

⬁

Ý aj h j Ž x . ,

Ž 10.60 .

js0

where h j Ž x . is the so-called normalized Hermite polynomial of degree j, namely, hj Ž x. s

1

Ž '2␲ j! .

1r2

eyx

2

r4

Hj Ž x . ,

j s 0, 1, . . . ,

⬁ ⬁ and a j s Hy⬁ f Ž x . h j Ž x . dx, since Hy⬁ h 2j Ž x . dxs 1 by virtue of Ž10.30.. Schwartz Ž1967. considered an estimate of f Ž x . of the form

qŽ n.

fˆn Ž x . s

Ý aˆjn h j Ž x . ,

js0

where a ˆjn s

1 n

n

Ý h j Ž Xk . ,

ks1

462

ORTHOGONAL POLYNOMIALS

and q Ž n. is a suitably chosen integer dependent on n such that q Ž n. s oŽ n. as n ™ ⬁. Under these conditions, Schwartz Ž1967, Theorem 1. showed that fˆnŽ x . is a consistent estimator of f Ž x . in the mean integrated squared error sense, that is, lim E

n™⬁

H

fˆn Ž x . y f Ž x .

2

dxs 0.

Under additional conditions on f Ž x ., fˆnŽ x . is also consistent in the mean squared error sense, that is, lim E f Ž x . y fˆn Ž x .

n™⬁

2

s0

uniformly in x. 10.9.2. Applications of Jacobi and Laguerre Polynomials Dasgupta Ž1968. presented an approximation to the distribution function of X s 12 Ž r q 1., where r is the sample correlation coefficient, in terms of a beta density and Jacobi polynomials. Similar methods were used by Durbin and Watson Ž1951. in deriving an approximation of the distribution of a statistic used for testing serial correlation in least-squares regression. Quadratic forms in random variables, which can often be regarded as having joint multivariate normal distributions, play an important role in analysis of variance and in estimation of variance components for a random or a mixed model. Approximation of the distributions of such quadratic forms can be carried out using Laguerre polynomials Žsee, for example, Gurland, 1953, and Johnson and Kotz, 1968.. Tiku Ž1964a. developed Laguerre series expansions of the distribution functions of the nonnormal variance ratios used for testing the homogeneity of treatment means in the case of one-way classification for analysis of variance with nonidentical group-to-group error distributions that are not assumed to be normal. Tiku Ž1964b. also used Laguerre polynomials to obtain an approximation to the first negative moment of a Poisson random variable, that is, the value of E Ž Xy1 ., where X has the Poisson distribution. More recently, Schone ¨ and Schmid Ž2000. made use of Laguerre polynomials to develop a series representation of the joint density and the joint distribution of a quadratic form and a linear form in normal variables. Such a representation can be used to calculate, for example, the joint density and the joint distribution function of the sample mean and sample variance. Note that for autocorrelated variables, the sample mean and sample variance are, in general, not independent. 10.9.3. Calculation of Hypergeometric Probabilities Using Discrete Chebyshev Polynomials The hypergeometric distribution is a discrete distribution, somewhat related to the binomial distribution. Suppose, for example, we have a lot of M items,

463

APPLICATIONS IN STATISTICS

r of which are defective and My r of which are nondefective. Suppose that we choose at random m items without replacement from the lot Ž m F M .. Let X be the number of defectives found. Then, the probability that X s k is given by

PŽ Xsk. s

ž /ž / ž / r k

My r myk , M m

Ž 10.61 .

where maxŽ0, m y Mq r . F k F minŽ m, r .. A random variable with the probability mass function Ž10.61. is said to have a hypergeometric distribution. We denote such a probability function by hŽ k; m, r, M .. There are tables for computing the probability value in Ž10.61. Žsee,for example, the tables given by Lieberman and Owen, 1961.. There are also several algorithms for computing this probability. Recently, Alvo and Cabilio Ž2000. proposed to represent the hypergeometric distribution in terms of discrete Chebyshev polynomials, as was seen in Section 10.8. The following is m a summary of this work: Consider the sequence
t nŽ k, m.4ns0 of discrete Chebyshev polynomials defined over the set of integers k s 0, 1, 2, . . . , m wsee formula Ž10.42.x, which is given by n

tn Ž k , m. s

Ý Ž y1. i

is0

ž /ž / n i

n q i k! Ž m y i . ! , i Ž k y i . !m!

n s 0, 1, . . . , m,

k s 0, 1, . . . , m.

Ž 10.62 .

Let X have the hypergeometric distribution as in Ž10.61.. Then according to Theorem 1 in Alvo and Cabilio Ž2000., m

Ý t n Ž k , m . h Ž k ; m, r , M . s t n Ž r , M .

Ž 10.63 .

ks0

for all n s 0, 1, . . . , m and r s 0, 1, . . . , M. Let t n s w t n Ž0, m ., t nŽ1, m., . . . , t nŽ m, m.xX , n s 0, 1, . . . , m, be the base vectors in an Ž m q 1.dimensional Euclidean space determined from the Chebyshev polynomials. Let g Ž k . be any function defined over the set of integers, k s 0, 1, . . . , m. Then g Ž k . can be expressed as gŽ k. s

m

Ý

g n tn Ž k , m . ,

k s 0, 1, . . . , m,

Ž 10.64 .

ns0

where g n s g ⭈ t nr5t n 5 2 , and g s w g Ž0., g Ž1., . . . , g Ž m.xX . Now, using the result

464

ORTHOGONAL POLYNOMIALS

in Ž10.63., the expected value of g Ž X . is given by E gŽ X . s

m

Ý g Ž k . h Ž k ; m, r , M .

ks0 m

s

m

Ý Ý

g n t n Ž k , m . h Ž k ; m, r , M .

ks0 ns0 m

s

Ý

ns0 m

s

Ý

m

Ý t n Ž k, m . h Ž k ; m, r , M .

gn

ks0

g n tn Ž r , M . .

Ž 10.65 .

ns0

This shows that the expected value of g Ž X . can be computed from knowledge of the coefficients g n and the discrete Chebyshev polynomials up to order m, evaluated at r and M. In particular, if g Ž x . is an indicator function taking the value one at x s k and the value zero elsewhere, then E g Ž X . sPŽ Xsk. s h Ž k ; m, r , M . . Applying the result in Ž10.65., we then obtain h Ž k ; m, r , M . s

m

tn Ž k , m .

ns0

tn

Ý

2

tn Ž r , M . .

Ž 10.66 .

Because of the recurrence relation Ž10.43. for discrete Chebyshev polynomials, calculating the hypergeometric probability using Ž10.66. can be done simply on a computer.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Abramowitz, M., and I. A. Stegun Ž1964.. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Wiley, New York. ŽThis useful volume was prepared by the National Bureau of Standards. It was edited by Milton Abramowitz and Irene A. Stegun.. Alvo, M., and P. Cabilio Ž2000.. ‘‘Calculation of hypergeometric probabilities using Chebyshev polynomials.’’ Amer. Statist., 54, 141᎐144. Cheney, E. W. Ž1982.. Introduction to Approximation Theory, 2nd ed. Chelsea, New York. ŽLeast-squares polynomial approximation is discussed in Chap. 4..

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

465

Chihara, T. S. Ž1978.. An Introduction to Orthogonal Polynomials. Gordon and Breach, New York. ŽThis text deals with the general theory of orthogonal polynomials, including recurrence relations, and some particular systems of orthogonal polynomials.. Cornish, E. A., and R. A. Fisher Ž1937.. ‘‘Moments and cumulants in the specification of distributions.’’ Re®. Internat. Statist. Inst., 5, 307᎐320. Cramer, ´ H. Ž1946.. Mathematical Methods of Statistics. Princeton University Press, Princeton. ŽThis classic book provides the mathematical foundation of statistics. Chap. 17 is a good source for approximation of density functions. . Dasgupta, P. Ž1968.. ‘‘An approximation to the distribution of sample correlation coefficient, when the population is non-normal.’’ Sankhya, Ser. B., 30, 425᎐428. Davis, P. J. Ž1975.. Interpolation and Approximation. Dover, New York. ŽChaps. 8 and 10 discuss least-squares approximation and orthogonal polynomials.. Divgi, D. R. Ž1979.. ‘‘Calculation of univariate and bivariate normal probability functions.’’ Ann. Statist., 7, 903᎐910. Durbin, J., and G. S. Watson Ž1951.. ‘‘Testing for serial correlation in least-squares regression II.’’ Biometrika, 38, 159᎐178. Freud, G. Ž1971.. Orthogonal Polynomials. Pergamon Press, Oxford. ŽThis book deals with fundamental properties of orthogonal polynomials, including Legendre, Chebyshev, and Jacobi polynomials. Convergence theory of series of orthogonal polynomials is discussed in Chap. 4.. Gurland, J. Ž1953.. ‘‘Distributions of quadratic forms and ratios of quadratic forms.’’ Ann. Math. Statist., 24, 416᎐427. Jackson, D. Ž1941.. Fourier Series and Orthogonal Polynomials. Mathematical Association of America. ŽThis classic monograph provides a good coverage of orthogonal polynomials, including Legendre, Jacobi, Hermite, and Laguerre polynomials. The presentation is informative and easy to follow.. Johnson, N. L., and S. Kotz Ž1968.. ‘‘Tables of distributions of positive definite quadratic forms in central normal variables.’’ Sankhya, Ser. B, 30, 303᎐314. Johnson, N. L., and S. Kotz Ž1970.. Continuous Uni®ariate Distributionsᎏ1. Houghton Mifflin, Boston. ŽChap. 12 contains a good discussion concerning the Cornish᎐Fisher expansion of quantiles. . Kendall, M. G., and A. Stuart Ž1977.. The Ad®anced Theory of Statistics, Vol. 1, 4th ed. Macmillan, New York. ŽThis classic book provides a good source for learning about the Gram᎐Charlier series of Type A and the Cornish᎐Fisher expansion. . Kerridge, D. F., and G. W. Cook Ž1976.. ‘‘Yet another series for the normal integral.’’ Biometrika, 63, 401᎐403. Lieberman, G. J., and Owen, D. B. Ž1961.. Tables of the Hypergeometric Probability Distribution. Stanford University Press, Palo Alto, California. Ralston, A., and P. Rabinowitz Ž1978.. A First Course in Numerical Analysis. McGraw-Hill, New York. ŽChap. 7 discusses Chebyshev polynomials of the first kind.. Rivlin, T. Ž1990.. Chebyshe® Polynomials, 2nd ed. Wiley, New York. ŽThis book gives a survey of the most important properties of Chebyshev polynomials.. Schone, ¨ A., and W. Schmid Ž2000.. ‘‘On the joint distribution of a quadratic and a linear form in normal variables.’’ J. Mult. Analysis, 72, 163᎐182.

466

ORTHOGONAL POLYNOMIALS

Schwartz, S. C. Ž1967.. ‘‘Estimation of probability density by an orthogonal series,’’ Ann. Math. Statist., 38, 1261᎐1265. Subrahmaniam, K. Ž1966.. ‘‘Some contributions to the theory of non-normalityᎏI Žunivariate case..’’ Sankhya, Ser. A, 28, 389᎐406. Szego, ¨ G. Ž1975.. Orthogonal Polynomials, 4th ed. Amer. Math. Soc., Providence, Rhode Island. ŽThis much-referenced book provides a thorough coverage of orthogonal polynomials.. Tiku, M. L. Ž1964a.. ‘‘Approximating the general non-normal variance ratio sampling distributions.’’ Biometrika, 51, 83᎐95. Tiku, M. L. Ž1964b.. ‘‘A note on the negative moments of a truncated Poisson variate.’’ J. Amer. Statist. Assoc., 59, 1220᎐1224. Viskov, O. V. Ž1992.. ‘‘Some remarks on Hermite polynomials.’’ Theory of Probability and its Applications, 36, 633᎐637.

EXERCISES In Mathematics 10.1. Show that the sequence
1r '2 , cos x, sin x, cos 2 x, sin 2 x, . . . , cos nx, sin nx, . . . 4 is orthonormal with respect to w Ž x . s 1r␲ over wy␲ , ␲ x. 10.2. Let
pnŽ x .4⬁ns0 be a sequence of Legendre polynomials (a) Use the Rodrigues formula to show that 1 (i) Hy1 x m pnŽ x . dxs 0 for m s 0, 1, . . . , n y 1, 2 nq1 2 n y1 1 (ii) Hy1 x n pnŽ x . dxs , n s 0, 1, 2, . . . 2nq1 n 1 (b) Deduce from Ža. that Hy1 pnŽ x .␲ny1 Ž x . dxs 0, where ␲ny1 Ž x . denotes an arbitrary polynomial of degree at most equal to n y 1. 1 (c) Make use of Ža. and Žb. to show that Hy1 pn2 Ž x . dx s 2rŽ2 n q 14 , n s 0, 1, . . . .

ž /

10.3. Let
TnŽ x .4⬁ns0 be a sequence of Chebyshev polynomials of the first kind. Let ␨ i s coswŽ2 i y 1.␲r2 n x, i s 1, 2, . . . , n. (a) Verify that ␨ 1 , ␨ 2 , . . . , ␨n are zeros of TnŽ x ., that is, TnŽ ␨ i . s 0, i s 1, 2, . . . , n. (b) Show that ␨ 1 , ␨ 2 , . . . , ␨n are simple zeros of TnŽ x .. w Hint: show that TnX Ž ␨ i . / 0 for i s 1, 2, . . . , n.x 10.4. Let
HnŽ x .4⬁ns0 be a sequence of Hermite polynomials. Show that dHn Ž x . (a) s nHny1 Ž x ., dx d 2 Hn Ž x . dHn Ž x . (b) yx q nHnŽ x . s 0. 2 dx dx

467

EXERCISES

10.5. Let
TnŽ x .4⬁ns0 and
UnŽ x .4⬁ns0 be sequences of Chebyshev polynomials of the first and second kinds, respectively, (a) Show that < UnŽ x .< F n q 1 for y1 F xF 1. w Hint: Use the representation Ž10.18. and mathematical induction on n.x (b) Show that < dTnŽ x .rdx < F n2 for all y1 F x F 1, with equality holding only if x s .1 Ž n G 2.. (c) Show that Tn Ž t .

Hy1 '1 y t x

dts y

2

Uny1 Ž x . n

'1 y x 2

for y1 F xF 1 and n / 0. 10.6. Show that the Laguerre polynomial L nŽ x . of degree n satisfies the differential equation x

d 2 Ln Ž x . dx

2

q Ž ␣q1yx.

dLn Ž x . dx

q nL n Ž x . s 0.

10.7. Consider the function H Ž x, t . s

1

Ž1yt .

␣q1

eyx t rŽ1yt . ,

␣ ) y1.

Expand H Ž x, t . as a power series in t and let the coefficient of t n be denoted by Žy1. n g nŽ x .rn! so that H Ž x, t . s

⬁

Ý

Ž y1.

ns0

n!

n

gnŽ x . t n .

Show that g nŽ x . is identical to L nŽ x . for all n, where L nŽ x . is the Laguerre polynomial of degree n. 10.8. Find the least-squares polynomial approximation of the function f Ž x . s e x over the interval wy1, 1x by using a Legendre polynomial of degree not exceeding 4. 10.9. A function f Ž x . defined on y1 F xF 1 can be represented using an infinite series of Chebyshev polynomials of the first kind, namely, f Ž x. s

c0 2

q

⬁

Ý c nTn Ž x . ,

ns1

468

ORTHOGONAL POLYNOMIALS

where cn s

2

f Ž x . Tn Ž x .

H ␲ y1 '1 y x 1

2

dx,

n s 0, 1, . . . .

This series converges uniformly whenever f Ž x . is continuous and of bounded variation on wy1, 1x. Approximate the function f Ž x . s e x using the first five terms of the above series. In Statistics 10.10. Suppose that from a certain distribution with a mean equal to zero we have knowledge of the following central moments: ␮ 2 s 1.0, ␮ 3 s y0.91, ␮4 s 4.86, ␮5 s y12.57, ␮6 s 53.22. Obtain an approximation for the density function of the distribution using Gram᎐Charlier series of type A. 10.11. The Cornish᎐Fisher expansion for x␣ , the upper ␣-quantile of a certain distribution, standardized so that its mean and variance are equal to zero and one, respectively, is of the following form Žsee Johnson and Kotz, 1970, page 34.: x␣ s z␣ q 16 Ž z␣2 y 1 . ␬ 3 q 241 Ž z␣3 y 3 z␣ . ␬ 4 1 y 361 Ž 2 z␣3 y 5 z␣ . ␬ 32 q 120 Ž z␣4 y 6 z␣2 q 3 . ␬5 1 y 241 Ž z␣4 y 5 z␣2 q 2 . ␬ 3 ␬ 4 q 324 Ž 12 z␣4 y 53 z␣2 q 17 . ␬ 33 1 q 720 Ž z␣5 y 10 z␣3 q 15 z␣ . ␬6 1 y 180 Ž 2 z␣5 y 17z␣3 q 21 z␣ . ␬ 3 ␬5 1 y 384 Ž 3 z␣5 y 24 z␣3 q 29 z␣ . ␬ 42 1 q 288 Ž 14 z␣5 y 103 z␣3 q 107z␣ . ␬ 32␬ 4 1 y 7776 Ž 252 z␣5 y 1688 z␣3 q 1511 z␣ . ␬ 34 q ⭈⭈⭈ ,

where z␣ is the upper ␣-quantile of the standard normal distribution, and ␬ r is the the r th cumulant of the distribution Ž r s 3, 4, . . . .. Apply this expansion to finding the upper 0.05-quantile of the central chisquared distribution with n s 5 degrees of freedom. w Note: The mean and variance of a central chi-squared distribution with n degrees of freedom are n and 2 n, respectively. Its r th cumu-

469

EXERCISES

lant, denoted by ␬ rX , is

␬ rX s n Ž r y 1 . ! 2 ry1 ,

r s 1, 2, . . . .

Hence, the r th cumulant, ␬ r , of the standardized chi-squared distribution is ␬ r s Ž2 n.yr r2␬ rX Ž r s 2, 3, . . . ..x 10.12. The normal integral H0x eyt

H0 e x

yt 2 r2

dts xy

2

r2

dt can be calculated from the series

x3 2 ⭈ 3 ⭈ 1!

q

x5 2 2 ⭈ 5 ⭈ 2!

y

x7 2 3 ⭈ 7 ⭈ 3!

q ⭈⭈⭈ .

(a) Use this series to obtain an approximate value for H01 eyt r2 dt. (b) Redo part Ža. using the series given by formula Ž10.59., that is, 2

H0 e x

yt 2 r2

dts xeyx

2

r8

⬁

Ý

⌰ 2 n Ž xr2.

ns0

2nq1

.

(c) Compare the results from Ža. and Žb. with regard to the number of terms in each series needed to achieve an answer correct to five decimal places. 10.13. Show that the expansion given by formula Ž10.46. is equivalent to representing the density function g Ž x . as a series of the form gŽ x. s

c n d n␾ Ž x .

⬁

Ý

ns0

n!

dx n

,

where ␾ Ž x . is the standard normal density function, and the c n’s are constant coefficients . 10.14. Consider the random variable n

Ws

Ý X i2 ,

is1

where X 1 , X 2 , . . . , X n are independent random variables from a distribution with the density function f Ž x. s␾ Ž x. y

␭3 d 3␾ Ž x . 6

dx 3

q

␭4 d 4␾ Ž x . 24

dx 4

q

␭23 d 6␾ Ž x . 72

dx 6

,

470

ORTHOGONAL POLYNOMIALS

where ␾ Ž x . is the standard normal density function and the quantities ␭3 and ␭4 are, respectively, the standard measures of skewness and kurtosis for the distribution. Obtain the moment generating function of W, and compare it with the moment generating function of a chi-squared distribution with n degrees of freedom. ŽSee Example 6.9.8 in Section 6.9.3.. w Hint: Use Hermite polynomials.x 10.15. A lot of Ms 10 articles contains r s 3 defectives and 7 good articles. Suppose that a sample of m s 4 articles is drawn from the lot without replacement. Let X denote the number of defective articles in the sample. Find the expected value of g Ž X . s X 3 using formula Ž10.65..

CHAPTER 11

Fourier Series

Fourier series were first formalized by the French mathematician JeanBaptiste Joseph Fourier Ž1768᎐1830. as a result of his work on solving a particular partial differential equation known as the heat conduction equation. However, the actual introduction of the so-called Fourier theory was motivated by a problem in musical acoustics concerning vibrating strings. Daniel Bernoulli Ž1700᎐1782. is credited as being the first to model the motion of a vibrating string as a series of trigonometric functions in 1748, twenty years before the birth of Fourier. The actual development of Fourier theory took place in 1807 upon Fourier’s return from Egypt, where he was a participant in the Egyptian campaign of 1798 under Napoleon Bonaparte.

11.1. INTRODUCTION A series of the form a0

q

2

⬁

Ý w an cos nxq bn sin nx x

Ž 11.1 .

ns1

is called a trigonometric series. Let f Ž x . be a function defined and Riemann integrable on the interval wy␲ , ␲ x. By definition, the Fourier series associated with f Ž x . is a trigonometric series of the form Ž11.1., where a n and bn are given by

an s bn s

1

␲

1

␲

H f Ž x . cos nx dx, ␲ y␲ H f Ž x . sin nx dx, ␲ y␲

n s 0, 1, 2, . . . ,

Ž 11.2 .

n s 1, 2, . . . .

Ž 11.3 . 471

472

FOURIER SERIES

In this case, we write a0

f Ž x. ;

q

2

⬁

Ý w an cos nxq bn sin nx x .

Ž 11.4 .

ns1

The numbers a n and bn are called the Fourier coefficients of f Ž x .. The symbol ; is used here instead of equality because at this stage, nothing is known about the convergence of the series in Ž11.4. for all x in wy␲ , ␲ x. Even if the series converges, it may not converge to f Ž x .. We can also consider the following reverse approach: if the trigonometric series in Ž11.4. is uniformly convergent to f Ž x . on wy␲ , ␲ x, that is, a0

f Ž x. s

q

2

⬁

Ý w an cos nxq bn sin nx x ,

Ž 11.5 .

ns1

then a n and bn are given by formulas Ž11.2. and Ž11.3.. In this case, the derivation of a n and bn is obtained by multiplying both sides of Ž11.5. by cos nx and sin nx, respectively, followed by integration over wy␲ , ␲ x. More specifically, to show formula Ž11.2., we multiply both sides of Ž11.5. by cos nx. For n / 0, we then have f Ž x . cos nxs

a0 2

⬁

Ý w ak cos kx cos nxq bk sin kx cos nx x dx. Ž 11.6 .

cos nxq

ks1

Since the series on the right-hand side converges uniformly, it can be integrated term by term Žthis can be easily proved by applying Theorem 6.6.1 to the sequence whose nth term is the nth partial sum of the series.. We then have a0

␲

q

Ý

␲

Hy␲ f Ž x . cos nx dxs 2 Hy␲ cos nx dx ⬁

ks1

␲

Hy␲a

k

cos kx cos nx dxq

␲

Hy␲ b

k

sin kx cos nx dx .

Ž 11.7 . But ␲

Hy␲cos nx dxs 0,

n s 1, 2, . . . ,

␲

Hy␲sin kx cos nx dxs 0,

Hy␲cos kx cos nx dxs ½ 0,␲ , ␲

Ž 11.8 . Ž 11.9 .

k / n, k s n G 1.

Ž 11.10 .

473

INTRODUCTION

From Ž11.7. ᎐ Ž11.10. we conclude Ž11.2.. Note that formulas Ž11.9. and Ž11.10. can be shown to be true by recalling the following trigonometric identities: sin kx cos nxs 12
sin Ž k q n . x q sin Ž k y n . x

4,

cos kx cos nxs 12
cos Ž k q n . x q cos Ž k y n . x

4.

For n s 0 we obtain from Ž11.7., ␲

Hy␲ f Ž x . dxs a ␲ . 0

Formula Ž11.3. for bn can be proved similarly. We can therefore state the following conclusion: a uniformly convergent trigonometric series is the Fourier series of its sum. Note. If the series in Ž11.1. converges or diverges at a point x 0 , then it converges or diverges at x 0 q 2 n␲ Ž n s 1, 2, . . . . due to the periodic nature of the sine and cosine functions. Thus, if the series Ž11.1. represents a function f Ž x . on wy␲ , ␲ x, then the series also represents the so-called periodic extension of f Ž x . for all values of x. Geometrically speaking, the periodic extension of f Ž x . is obtained by shifting the graph of f Ž x . on wy␲ , ␲ x by 2␲ , 4␲ , . . . to the right and to the left. For example, for y3␲ - xF y␲ , f Ž x . is defined by f Ž xq 2␲ ., and for ␲ - xF 3␲ , f Ž x . is defined by f Ž xy 2␲ ., etc. This defines f Ž x . for all x as a periodic function with period 2␲ . EXAMPLE 11.1.1.

Let f Ž x . be defined on wy␲ , ␲ x by the formula

° 0, ¢␲

y␲ F x- 0,

~x,

f Ž x. s

0 F xF ␲ .

Then, from Ž11.2. we have an s s

s

s

1

␲

H f Ž x . cos nx dx ␲ y␲ 1

␲

␲

2

␲

x sin nx

1

␲

H0 x cos nx dx

2

1

␲ 2 n2

y

n

0 n

1

␲

H sin nx dx n 0

Ž y1. y 1 .

474

FOURIER SERIES

Thus, for n s 1, 2, . . . , an s

½

n even, n odd.

0, y2r Ž ␲ 2 n2 . ,

Also, from Ž11.3., we get bn s s s

s

s

1

␲

H f Ž x . sin nx dx ␲ y␲ 1

␲

␲

2

H0 x sin nx dx

1

␲

y

2

1

y

␲2

Ž y1.

x cos nx

␲

n

0

␲ n

q

n

Ž y1. q

1

␲

H cos nx dx n 0

sin nx n2

␲ 0

nq1

.

␲n

For n s 0, a0 is given by a0 s s

1

␲

H f Ž x . dx ␲ y␲ 1

␲

␲

2

H0 x dx

s 12 . The Fourier series of f Ž x . is then of the form f Ž x. ;

1 4

y

2

␲2

EXAMPLE 11.1.2.

⬁

1

ns1

Ž 2 n y 1.

Ý

2

cos Ž 2 n y 1 . x y

⬁

1

␲

Let f Ž x . s x 2 Žy␲ F xF ␲ .. Then a0 s s

1

␲

H x ␲ y␲ 2␲ 2 3

,

2

dx

Ý

ns1

Ž y1. n

n

sin nx.

475

CONVERGENCE OF FOURIER SERIES

an s s

1

␲

H x ␲ y␲

s

s bn s

cos nx dx ␲

x 2 sin nx

y

␲n

sy

s

2

y␲

2

2

␲

H x sin nx dx ␲ n y␲

␲

H x sin nx dx ␲ n y␲

2 x cos nx

␲n

2

␲

y y␲

2

␲n

␲

2

Hy␲cos nx dx

4 cos n␲ n2 4 Ž y1 .

n

,

n2 1

␲

H x ␲ y␲

2

n s 1, 2, . . . ,

sin nx dx

s 0, since x 2 sin nx is an odd function. Thus, the Fourier expansion of f Ž x . is x ; 2

␲2 3

ž

y 4 cos xy

cos 2 x 22

q

cos 3 x 32

/

y ⭈⭈⭈ .

11.2. CONVERGENCE OF FOURIER SERIES In this section, we consider the conditions under which the Fourier series of f Ž x . converges to f Ž x .. We shall assume that f Ž x . is Riemann integrable on wy␲ , ␲ x. Hence, f 2 Ž x . is also Riemann integrable on wy␲ , ␲ x by Corollary 6.4.1. This condition will be satisfied if, for example, f Ž x . is continuous on wy␲ , ␲ x, or if it has a finite number of discontinuities of the first kind Žsee Definition 3.4.2. in this interval. In order to study the convergence of Fourier series, the following lemmas are needed: Lemma 11.2.1.

If f Ž x . is Riemann integrable on wy␲ , ␲ x, then ␲

H f Ž x . cos nx dxs 0, n™⬁ y␲ lim

␲

H f Ž x . sin nx dxs 0. n™⬁ y␲ lim

Ž 11.11 . Ž 11.12 .

476

FOURIER SERIES

Proof. Let snŽ x . denote the following partial sum of Fourier series of f Ž x .: a0

sn Ž x . s

n

Ý w ak cos kxq bk sin kx x ,

q

2

Ž 11.13 .

ks1

where a k Ž k s 0, 1, . . . , n. and bk Ž k s 1, 2, . . . , n. are given by Ž11.2. and Ž11.3., respectively. Then ␲

a0

␲

q

Ý

Hy␲ f Ž x . s Ž x . dxs 2 Hy␲ f Ž x . dx n

n

ak

ks1

s

␲ a20 2

␲

␲

Hy␲ f Ž x . cos kx dxq b Hy␲ f Ž x . sin kx dx k

n

q␲

Ý Ž a2k q bk2 . .

ks1

It can also be verified that ␲

Hy␲

sn2 Ž x . dxs

␲ a20 2

n

q␲

Ý Ž a2k q bk2 . .

ks1

Consequently, ␲

Hy␲

f Ž x . y sn Ž x .

2

dxs s

␲

Hy␲ f ␲

Hy␲ f

2

␲

␲

Ž x . dxy 2H f Ž x . sn Ž x . dxq H sn2 Ž x . dx y␲

2

␲ a20

Ž x . dxy

2

y␲

n

q␲

Ý Ž a2k q bk2 .

.

ks1

It follows that a20 2

n

q

1

␲

Ý Ž a2k q bk2 . F ␲ H

y␲

ks1

f 2 Ž x . dx.

Ž 11.14 .

Since the right-hand side of Ž11.14. exists and is independent of n, the series Ý⬁ks1 Ž a2k q bk2 . must be convergent. This follows from applying Theorem 5.1.2 and the fact that the sequence sUn s

n

Ý Ž a2k q bk2 .

ks1

is bounded and monotone increasing. But, the convergence of Ý⬁ks1 Ž a2k q bk2 . implies that lim k ™⬁Ž a2k q bk2 . s 0 Žsee Result 5.2.1 in Chapter 5.. Hence, lim k ™⬁ a k s 0, lim k ™⬁ bk s 0. I

477

CONVERGENCE OF FOURIER SERIES

Corollary 11.2.1.

If ␾ Ž x . is Riemann integrable on wy␲ , ␲ x,then ␲

H ␾ Ž x . sin Ž n q . x n™⬁ y␲ lim

1 2

Ž 11.15 .

dxs 0.

Proof. We have that

␾ Ž x . sin Ž n q 12 . x s ␾ Ž x . cos

x 2

sin nxq ␾ Ž x . sin

x 2

cos nx.

Let ␾ 1Ž x . s ␾ Ž x .sinŽ xr2., ␾ 2 Ž x . s ␾ Ž x .cosŽ xr2.. Both ␾ 1Ž x . and ␾ 2 Ž x . are Riemann integrable by Corollary 6.4.2. By applying Lemma 11.2.1 to both ␾ 1Ž x . and ␾ 2 Ž x ., we obtain ␲

H ␾ Ž x . cos nx dxs 0, n™⬁ y␲ lim

Ž 11.16 .

1

␲

H ␾ n™⬁ y␲ lim

2

Ž x . sin nx dxs 0.

Ž 11.17 .

Formula Ž11.15. follows from the addition of Ž11.16. and Ž11.17.. Corollary 11.2.2.

I

If ␾ Ž x . is Riemann integrable on wy␲ , ␲ x, then 0 ␾ Ž x . sin Ž n q . x H n™⬁ y␲ 1 2

lim

␲

H ␾ Ž x . sin Ž n q . x n™⬁ 0 lim

1 2

dxs 0, dxs 0.

Proof. Define the functions h1Ž x . and h 2 Ž x . as h1 Ž x . s h2 Ž x . s

½ ½

0, ␾Ž x. ,

0 F xF ␲ , y␲ F x- 0,

␾Ž x. , 0,

0 F xF ␲ , y␲ F x- 0.

Both h1Ž x . and h 2 Ž x . are Riemann integrable on wy␲ , ␲ x. Hence, by Corollary 11.2.1, 0 ␾ Ž x . sin Ž n q . x H n™⬁ y␲ 1 2

lim

␲

H h Ž x . sin Ž n q . x n™⬁ y␲

dxs lim

1

1 2

dx

s0 ␲

H ␾ Ž x . sin Ž n q . x n™⬁ 0 lim

1 2

␲

H h Ž x . sin Ž n q . x n™⬁ y␲

dxs lim s 0.

2

I

1 2

dx

478

FOURIER SERIES

Lemma 11.2.2. 1 2

n

q

Ý cos kus

ks1

sin

Ž n q 12 . u

2 sin Ž ur2.

Ž 11.18 .

.

Proof. Let GnŽ u. be defined as Gn Ž u . s

1

n

q

2

Ý cos ku.

ks1

Multiplying both sides by 2 sinŽ ur2. and using the identity 2 sin

u 2

Ž k q 12 . u

cos kus sin

y sin

Ž k y 12 . u

,

we obtain 2 sin

u 2

Gn Ž u . s sin

u

n

q

2

Ý
sin Ž k q 12 . u

Ž n q 12 . u

s sin

Ž k y 12 . u 4

y sin

ks1

.

Hence, if sinŽ ur2. / 0, then Gn Ž u . s

sin

Ž n q 12 . u

2sin Ž ur2.

I

.

Theorem 11.2.1. Let f Ž x . be Riemann integrable on wy␲ , ␲ x, and let it be extended periodically outside this interval. Suppose that at a point x, f Ž x . satisfies the following two conditions: i. Both f Ž xy. and f Ž xq. exist, where f Ž xy. and f Ž xq. are the left-sided and right-sided limits of f Ž x ., and f Ž x . s 12 f Ž xy . q f Ž xq . .

Ž 11.19 .

ii. Both one-sided derivatives, f X Ž xq . s limq h™0

f X Ž xy . s limy h™0

exist.

Ž xq h . y f Ž xq . h

,

f Ž xq h . y f Ž xy . h

,

479

CONVERGENCE OF FOURIER SERIES

Then the Fourier series of f Ž x . converges to f Ž x . at x, that is, sn Ž x . ™

½

f Ž x. 1 2

if x is a point of continuity,

w f Ž x . qf Ž x .x q

y

if x is a point of discontinuity of the first kind.

Before proving this theorem, it should be noted that if f Ž x . is continuous at x, then condition Ž11.19. is satisfied. If, however, x is a point of discontinuity of f Ž x . of the first kind, then f Ž x . is defined to be equal to the right-hand side of Ž11.19.. Such a definition of f Ž x . does not affect the values of the Fourier coefficients, a n and bn , in Ž11.2. and Ž11.3.. Hence, the Fourier series of f Ž x . remains unchanged. Proof of Theorem 11.2.1. From Ž11.2., Ž11.3., and Ž11.13., we have sn Ž x . s

1

n

␲

H f Ž t . dtq Ý 2␲ y␲

ž

1

ks1

␲ q

s

s

1

␲

H f Ž t. ␲ y␲ 1

␲

H f Ž t. ␲ y␲

␲

Hy␲ f Ž t . cos kt dt

1

␲

ž

/

cos kx

␲

Hy␲ f Ž t . sin kt dt

/

sin kx

n

1 2

Ý Ž cos kt cos kxq sin kt sin kx .

q

dt

ks1 n

1 2

q

Ý cos k Ž t y x .

dt.

ks1

Using Lemma 11.2.2, snŽ x . can be written as sn Ž x . s

1

␲

Hy␲

␲

f Ž t.

½

sin

Ž n q 12 . Ž t y x .

2 sin Ž t y x . r2

5

dt.

Ž 11.20 .

If we make the change of variable t y xs u in Ž11.20., we obtain sn Ž x . s

1

␲yx

␲

Hy␲yx

f Ž xq u .

sin

Ž n q 12 . u

2 sin Ž ur2.

du.

Since both f Ž xq u. and sinwŽ n q 12 . u xrw2sinŽ ur2.x have period 2␲ with respect to u, the integral from y␲ y x to ␲ y x has the same value as the one from y␲ to ␲ . Thus, sn Ž x . s

1

␲

H f Ž xq u . ␲ y␲

sin

Ž n q 12 . u

2sin Ž ur2.

du.

Ž 11.21 .

480

FOURIER SERIES

We now need to show that for each x, lim sn Ž x . s 12 f Ž xy . q f Ž xq . .

n™⬁

Formula Ž11.21. can be written as sn Ž x . s

s

1

H f Ž xq u . ␲ y␲ 0

1

␲

q

H ␲ 0

1

0

H ␲ y␲

1

Ž n q 12 . u

sin

␲

H ␲ 0

H ␲ y␲ 0

sin

Ž n q 12 . u

sin

1

␲

H ␲ 0

Ž n q 12 . u

sin

2 sin Ž ur2.

Ž n q 12 . u

du

du

2 sin Ž ur2.

sin

Ž n q 12 . u

2 sin Ž ur2.

f Ž xq u . y f Ž xq .

q f Ž xq .

du

2 sin Ž ur2.

f Ž xq u . y f Ž xy . 1

du

2 sin Ž ur2.

f Ž xq u .

q f Ž xy . q

Ž n q 12 . u

sin

du

Ž 11.22 .

du.

2 sin Ž ur2.

The first integral in Ž11.22. can be expressed as 1

H ␲ y␲ 0

s

f Ž xq u . y f Ž xy . 1

H ␲ y␲ 0

sin

Ž n q 12 . u

du

2 sin Ž ur2.

f Ž xq u . y f Ž xy .

u

u

2 sin Ž ur2.

sin

Ž n q 12 . u

du.

We note that the function f Ž xq u . y f Ž xy . u

⭈

u 2 sin Ž ur2.

is Riemann integrable on wy␲ , 0x, and at u s 0 it has a discontinuity of the first kind, since limy

f Ž xq u . y f Ž xy . u

u™0

limy

u™0

s f X Ž xy . ,

u 2 sin Ž ur2.

s 1,

and

481

CONVERGENCE OF FOURIER SERIES

that is, both limits are finite. Consequently, by applying Corollary 11.2.2 to the function
w f Ž xq u. y f Ž xy.x ru4 . urw2 sinŽ ur2.x, we get 1

lim

Hy␲ 0

␲

n™⬁

Ž n q 12 . u

sin

f Ž xq u . y f Ž x . y

2 sin Ž ur2.

dus 0.

Ž 11.23 .

We can similarly show that the third integral in Ž11.22. has a limit equal to zero as n ™ ⬁, that is, 1

lim

␲

n™⬁

␲

H0

sin

f Ž xq u . y f Ž xq .

Ž n q 12 . u

2 sin Ž ur2.

dus 0.

Ž 11.24 .

Furthermore, from Lemma 11.2.2, we have

Hy␲ 0

sin

Ž n q 12 . u

2 sin Ž ur2.

dus s

Hy␲ 0

␲ 2

␲

H0

sin

Ž n q 12 . u

2 sin Ž ur2.

dus s

␲

2

n

q

2

Ý cos ku

du

ks1

Ž 11.25 .

,

H0

␲

1

1 2

n

q

Ý cos ku

du

ks1

Ž 11.26 .

.

From Ž11.22. ᎐ Ž11.26., we conclude that lim sn Ž x . s 12 f Ž xy . q f Ž xq . .

n™⬁

I

Definition 11.2.1. A function f Ž x . is said to be piecewise continuous on w a, b x if it is continuous on w a, b x except for a finite number of discontinuities of the first kind in w a, b x, and, in addition, both f Ž aq. and f Ž by. exist. I Corollary 11.2.3. Suppose that f Ž x . is piecewise continuous on wy␲ , ␲ x, and that it can be extended periodically outside this interval. In addition, if, at each interior point of wy␲ , ␲ x, f X Ž xq. and f X Ž xy. exist and f X Žy␲q. and f X Ž␲y. exist, then at a point x, the Fourier series of f Ž x . converges to 1 w Ž y . q f Ž xq .x. 2 f x Proof. This follows directly from applying Theorem 11.2.1 and the fact that a piecewise continuous function on w a, b x is Riemann integrable there. I

482

FOURIER SERIES

EXAMPLE 11.2.1. Let f Ž x . s x Žy␲ F xF ␲ .. The periodic extension of f Ž x . Žoutside the interval wy␲ , ␲ x. is defined everywhere. In this case, an s bn s s

1

␲

1

␲

H x cos nx dxs 0, ␲ y␲

n s 0, 1, 2, . . . ,

H x sin nx dx ␲ y␲ 2 Ž y1 .

nq1

n s 1, 2, . . . .

,

n

Hence, the Fourier series of f Ž x . is ⬁

x;

Ý

2 Ž y1 .

nq1

sin nx.

n

ns1

This series converges to x at each x in Žy␲ , ␲ .. At xs y␲ , ␲ we have discontinuities of the first kind for the periodic extension of f Ž x .. Hence, at x s ␲ , the Fourier series converges to 1 2

f Ž ␲y . q f Ž ␲q . s 12 ␲ q Ž y␲ . s 0.

Similarly, at xs y␲ , the series converges to 1 2

f Ž y␲y . q f Ž y␲q . s 12 ␲ q Ž y␲ . s 0.

For other values of x, the series converges to the value of the periodic extension of f Ž x .. EXAMPLE 11.2.2. Consider the function f Ž x . s x 2 defined in Example 11.1.2. Its Fourier series is x ; 2

␲2 3

q4

⬁

Ý

ns1

Ž y1.

n

cos nx.

n2

The periodic extension of f Ž x . is continuous everywhere. We can therefore write x s 2

␲2 3

q4

⬁

Ý

ns1

Ž y1. n2

n

cos nx.

DIFFERENTIATION AND INTEGRATION OF FOURIER SERIES

483

In particular, for xs .␲ , we have

␲ s 2

␲2

s

6

␲2

⬁

q4

3

Ý

ns1

⬁

1

ns1

n2

Ý

Ž y1.

2n

n2

,

.

11.3. DIFFERENTIATION AND INTEGRATION OF FOURIER SERIES In Section 11.2 conditions were given under which a function f Ž x . defined on wy␲ , ␲ x is represented as a Fourier series. In this section, we discuss the conditions under which the series can be differentiated or integrated term by term. Theorem 11.3.1. Let a0r2 q Ý⬁ns1 w a n cos nxq bn sin nx x be the Fourier series of f Ž x .. If f Ž x . is continuous on wy␲ , ␲ x, f Žy␲ . s f Ž␲ ., and f X Ž x . is piecewise continuous on wy␲ , ␲ x, then a. at each point where f Y Ž x . exists, f X Ž x . can be represented by the derivative of the Fourier series of f Ž x ., where differentiation is done term by term, that is, fXŽ x. s

⬁

Ý w nbn cos nxy nan sin nx x ;

ns1

b. the Fourier series of f Ž x . converges uniformly and absolutely to f Ž x . on wy␲ , ␲ x. Proof. a. The Fourier series of f Ž x . converges to f Ž x . by Corollary 11.2.3. Thus, f Ž x. s

a0

q

2

⬁

Ý w an cos nxq bn sin nx x .

ns1

The periodic extension of f Ž x . is continuous, since f Ž␲ . s f Žy␲ .. Furthermore, the derivative, f X Ž x ., of f Ž x . satisfies the conditions of Corollary 11.2.3. Hence, the Fourier series of f X Ž x . converges to f X Ž x ., that is, fXŽ x. s

␣0 2

q

⬁

Ý w ␣ n cos nxq ␤n sin nx x ,

ns1

Ž 11.27 .

484

FOURIER SERIES

where

␣0 s s

1 1

s

1

␲

X

f Ž ␲ . y f Ž y␲ .

␲

s 0, 1 ␣n s ␲ s

␲

H f Ž x . dx ␲ y␲

␲

Hy␲ f Ž x . cos nx dx X

␲

f Ž x . cos nx y␲ q

Ž y1.

n

␲

H f Ž x . sin nx dx ␲ y␲

n

f Ž ␲ . y f Ž y␲ . q nbn

n

s nbn ,

␤n s s

1

␲

H f Ž x . sin nx dx ␲ y␲ 1

X

␲

f Ž x . sin nx y␲ y

␲ s ynan .

n

␲

H f Ž x . cos nx dx ␲ y␲

By substituting ␣ 0 , ␣ n , and ␤n in Ž11.27., we obtain fXŽ x. s

⬁

Ý w nbn cos nxy nan sin nx x .

ns1

b. Consider the Fourier series of f X Ž x . in Ž11.27., where ␣ 0 s 0, ␣ n s nbn , ␤n s ynan . Then, using inequality Ž11.14., we obtain ⬁

1

␲

Ý Ž ␣ n2 q ␤n2 . F ␲ H

y␲

ns1

fXŽ x.

2

dx.

Ž 11.28 .

Inequality Ž11.28. indicates that Ý⬁ns1 Ž ␣ n2 q ␤n2 . is a convergent series. Now, let snŽ x . s a0r2 q Ý nks1 w a k cos kx q b k sin kx x. Then, for n G m q 1, sn Ž x . y sm Ž x . s

n

Ý w ak cos kxq bk sin kx x

ksmq1 n

F

Ý

ksmq1

a k cos kxq bk sin kx .

485

DIFFERENTIATION AND INTEGRATION OF FOURIER SERIES

Note that a k cos kxq bk sin kx F Ž a2k q bk2 .

1r2

Ž 11.29 .

.

Inequality Ž11.29. follows from the fact that a k cos kxq bk sin kx is the dot product, u ⴢ v, of the vectors u s Ž a k , bk .X , v s Žcos kx, sin kx .X , and < u ⴢ v < F 5u 5 2 5v 5 2 Žsee Theorem 2.1.2.. Hence, n

Ý Ž a2k q bk2 .

sn Ž x . y sm Ž x . F

1r2

ksmq1 n

1

Ý

s

k

ksmq1

Ž ␣ k2 q ␤k2 .

1r2

.

Ž 11.30 .

But, by the Cauchy᎐Schwarz inequality, n

Ý

ksmq1

1 k

Ž ␣ k2 q ␤k2 .

1r2

n

1r2

1

Ý

F

Ž ␣ k2 q ␤k2 .

Ý

k2

ksmq1

n

ksmq1

1r2

,

and by Ž11.28., n

1

␲

Ý Ž ␣ k2 q ␤k2 . F ␲ H

y␲

ksmq1

fXŽ x.

2

dx.

In addition, Ý⬁ks1 1rk 2 is a convergent series. Hence, by the Cauchy criterion, for a given ⑀ ) 0, there exists a positive integer N such that Ý nksmq1 1rk 2 - ⑀ 2 if n ) m ) N. Hence, n

sn Ž x . y sm Ž x . F

Ý

a k cos kxq bk sin kx

ksmq1

F c⑀ ,

Ž 11.31 .

if m ) n ) N,

␲ w X Ž .x 2 dx41r2 . The double inequality Ž11.31. where c s
Ž1r␲ .Hy ␲ f x shows that the Fourier series of f Ž x . converges absolutely and uniI formly to f Ž x . on wy␲ , ␲ x by the Cauchy criterion.

Note that from Ž11.30. we can also conclude that Ý⬁ks1 Ž a2k q bk2 .1r2 satisfies the Cauchy criterion. This series is therefore convergent. Furthermore, it is easy to see that ⬁

ÝŽ

ks1

a k q bk . F

⬁

Ý

ks1

2 Ž a2k q bk2 .

1r2

.

486

FOURIER SERIES

This indicates that the series Ý⬁ks1 Ž< a k < q < bk < . is convergent by the comparison test. Note that, in general, we should not expect that a term-by-term differentiation of a Fourier series of f Ž x . will result in a Fourier series of f X Ž x .. For example, for the function f Ž x . s x, y␲ F xF ␲ , the Fourier series is Žsee Example 11.2.1. 2 Ž y1 .

⬁

x;

Ý

nq1

sin nx.

n

ns1

Differentiating this series term by term, we obtain Ý⬁ns1 2Žy1. nq1 cos nx. This, however, is not the Fourier series of f X Ž x . s 1, since the Fourier series of f X Ž x . s 1 is just 1. Note that in this case, f Ž␲ . / f Žy␲ ., which violates one of the conditions in Theorem 11.3.1. Theorem 11.3.2. Fourier series

If f Ž x . is piecewise continuous on wy␲ , ␲ x and has the

f Ž x. ;

a0

⬁

q

2

Ý w an cos nxq bn sin nx x ,

Ž 11.32 .

ns1

then a term-by-term integration of this series gives the Fourier series of x Hy⬁ f Ž t . dt for xg wy␲ , ␲ x, that is,

Hy␲ f Ž t . dts x

a0 Ž ␲ q x . 2

⬁

q

Ý

ns1

an n

sin nxy

bn n

Ž cos nxy cos n␲ . , y␲ F xF ␲ .

Furthermore, the integrated series converges uniformly to Hyx ␲ f Ž t . dt. Proof. Define the function g Ž x . as gŽ x. s

a0

Hy␲ f Ž t . dty 2 x. x

Ž 11.33 .

If f Ž x . is piecewise continuous on wy␲ , ␲ x, then it is Riemann integrable there, and by Theorem 6.4.7, g Ž x . is continuous on wy␲ , ␲ x. Furthermore, by Theorem 6.4.8, at each point where f Ž x . is continuous, g Ž x . is differentiable and gX Ž x . s f Ž x . y

a0 2

.

Ž 11.34 .

487

DIFFERENTIATION AND INTEGRATION OF FOURIER SERIES

This implies that g X Ž x . is piecewise continuous on wy␲ , ␲ x. In addition, g Žy␲ . s g Ž␲ .. To show this, we have from Ž11.33. g Ž y␲ . s s

y␲

Hy␲ a0

2

␲

a0

␲

Hy␲ f Ž t . dty 2 ␲

s a0 ␲ y s

a0

␲.

2 g Ž␲ . s

f Ž t . dtq

a0 2

a0 2

␲

␲,

by the definition of a0 . Thus, the function g Ž x . satisfies the conditions of Theorem 11.3.1. It follows that the Fourier series of g Ž x . converges uniformly to g Ž x . on wy␲ , ␲ x. We therefore have gŽ x. s

A0

q

2

⬁

Ý w A n cos nxq Bn sin nx x .

Ž 11.35 .

ns1

Moreover, by part Ža. of Theorem 11.3.1, we have gX Ž x . s

⬁

Ý w nBn cos nxy nA n sin nx x .

Ž 11.36 .

ns1

Then, from Ž11.32., Ž11.34., and Ž11.36., we obtain a n s nBn ,

n s 1, 2, . . . ,

bn s ynA n ,

n s 1, 2, . . . .

Substituting in Ž11.35., we get gŽ x. s

A0

q

2

⬁

Ý

y

ns1

bn n

cos nxq

an n

sin nx .

From Ž11.33. we then have

Hy␲ f Ž t . dts x

a0 x 2

q

A0 2

q

⬁

Ý

ns1

y

bn n

cos nxq

an n

sin nx .

Ž 11.37 .

488

FOURIER SERIES

To find the value of A 0 , we set xs y␲ in Ž11.37., which gives 0sy

a0 ␲

q

A0

2

Ý

ž

⬁

bn

⬁

q

2

ns1

y

bn n

/

cos n␲ .

Hence, A0

s

a0 ␲

2

Ý

q

2

ns1

n

cos n␲ .

Substituting A 0r2 in Ž11.37., we finally obtain

Hy␲ x

f Ž t . dts

a0 Ž ␲ q x .

q

2

⬁

an

Ý

ns1

n

sin nxy

bn n

Ž cos nxy cos n␲ . .

I

11.4. THE FOURIER INTEGRAL We have so far considered Fourier series corresponding to a function defined on the interval wy␲ , ␲ x. As was seen earlier in this chapter, if a function is initially defined on wy␲ , ␲ x, we can extend its definition outside wy␲ , ␲ x by considering its periodic extension. For example, if f Žy␲ . s f Ž␲ ., then we can define f Ž x . everywhere in Žy⬁, ⬁. by requiring that f Ž xq 2␲ . s f Ž x . for all x. The choice of the interval wy␲ , ␲ x was made mainly for convenience. More generally, we can now consider a function f Ž x . defined on the interval wyc, c x. For such a function, the corresponding Fourier series is given by a0

q

2

⬁

Ý

a n cos

ns1

ž / n␲ x c

q bn sin

ž / n␲ x c

,

Ž 11.38 .

where an s bn s

H f Ž x . cos ž c yc

n␲ x

1

n x

1

c

H f Ž x . sin c yc c

/ ␲ ž / c

c

dx,

n s 0, 1, 2, . . . ,

Ž 11.39 .

dx,

n s 1, 2, . . . .

Ž 11.40 .

Now, a question arises as to what to do when we have a function f Ž x . that is already defined everywhere on Žy⬁, ⬁., but is not periodic. We shall show that, under certain conditions, such a function can be represented by an infinite integral rather than by an infinite series. This integral is called a Fourier integral. We now show the development of such an integral. Substituting the expressions for a n and bn given by Ž11.39., Ž11.40. into Ž11.38., we obtain the Fourier series 1

1

⬁

H f Ž t . dtq c Ý Hyc f Ž t . cos 2 c yc c

ns1

c

n␲ c

Ž t y x . dt.

Ž 11.41 .

489

THE FOURIER INTEGRAL

If c is finite and f Ž x . satisfies the conditions of Corollary 11.2.3 on wyc, c x, then the Fourier series Ž11.41. converges to 12 w f Ž xy . q f Ž xq .x. However, this series representation of f Ž x . is not valid outside the interval wyc, c x unless f Ž x . is periodic with the period 2 c. In order to provide a representation that is valid for all values of x when f Ž x . is not periodic, we need to consider extending the series in Ž11.41. by letting c go to infinity, assuming that f Ž x . is absolutely integrable over the whole real line. We now show how this can be done: ⬁ As c ™ ⬁, the first term in Ž11.41. goes to zero provided that Hy⬁ f Ž t . dt exists. To investigate the limit of the series in Ž11.41. as c ™ ⬁, we set ␭1 s ␲rc, ␭2 s 2␲rc, . . . , ␭ n s n␲rc, . . . , ⌬ ␭ n s ␭ nq1 y ␭ n s ␲rc, n s 1, 2, . . . . We can then write 1 c

⬁

c

n␲

yc

c

Ý H f Ž t . cos

ns1

Ž t y x . dts

1

␲

⬁

Ý ⌬ ␭ nH f Ž t . cos

ns1

c

yc

␭ n Ž t y x . dt.

Ž 11.42 . When c is large, ⌬ ␭ n is small, and the right-hand side of Ž11.42. will be an approximation of the integral 1

⬁

½

⬁

H H f Ž t . cos ␲ 0 y⬁

5

␭ Ž t y x . dt d ␭ .

Ž 11.43 .

This is the Fourier integral of f Ž x .. Note that Ž11.43. can be written as ⬁

H0

a Ž ␭ . cos ␭ xq b Ž ␭ . sin ␭ x d ␭ ,

Ž 11.44 .

where aŽ ␭. s b Ž ␭. s

1

⬁

1

⬁

H f Ž t . cos ␭t dt, ␲ y⬁ H f Ž t . sin ␭t dt. ␲ y⬁

The expression in Ž11.44. resembles a Fourier series where the sum has been replaced by an integral and the parameter ␭ is used in place of the integer n. Moreover, aŽ ␭. and bŽ ␭. act like Fourier coefficients. We now show that the Fourier integral in Ž11.43. provides a representation for f Ž x . provided that f Ž x . satisfies the conditions of the next theorem. Theorem 11.4.1. Let f Ž x . be piecewise continuous on every finite inter⬁ < Ž .< val w a, b x. If Hy⬁ f x dx exists, then at every point x Žy⬁ - x - ⬁. where

490

FOURIER SERIES

f X Ž xq . and f X Ž xy . exist, the Fourier integral of f Ž x . converges to 12 w f Ž xy . q f Ž xq.x , that is, 1

⬁

½

⬁

H H f Ž t . cos ␲ 0 y⬁

5

␭Ž t y x . dt d ␭ s 12 f Ž xy . q f Ž xq . .

The proof of this theorem depends on the following lemmas: Lemma 11.4.1.

If f Ž x . is piecewise continuous on w a, b x, then

H n™⬁ a

b

f Ž x . sin nx dxs 0,

Ž 11.45 .

Ha

b

f Ž x . cos nx dxs 0.

Ž 11.46 .

lim

lim

n™⬁

Proof. Let the interval w a, b x be divided into a finite number of subintervals on each of which f Ž x . is continuous. Let any one of these subintervals be denoted by w p, q x. To prove formula Ž11.45. we only need to show that

H f Ž x . sin nx dxs 0. n™⬁ p q

Ž 11.47 .

lim

For this purpose, we divide the interval w p, q x into k equal subintervals using the partition points x 0 s p, x 1 , x 2 , . . . , x k s q. We can then write the integral in Ž11.47. as ky1

ÝH

is0

x iq1

f Ž x . sin nx dx,

xi

or equivalently as

½

ky1

Ý

is0

f Ž xi .

Hx

x iq1

sin nx dxq

i

Hx

5

x iq1

f Ž x . y f Ž x i . sin nx dx .

i

It follows that

Hp

q

f Ž x . sin nx dx F

ky1

Ý

f Ž xi .

cos nx i y cos nx iq1

is0 ky1

q

ÝH

is0

x iq1

xi

n f Ž x . y f Ž x i . dx.

491

THE FOURIER INTEGRAL

Let M denote the maximum value of < f Ž x .< on w p, q x. Then

Hp f Ž x . sin nx dx F q

2 Mk n

ky1

q

ÝH

x iq1

xi

is0

f Ž x . y f Ž x i . dx.

Ž 11.48 .

Furthermore, since f Ž x . is continuous on w p, q x, it is uniformly continuous there wif necessary, f Ž x . can be made continuous at p, q by simply using f Ž pq. and f Ž qy. as the values of f Ž x . at p, q, respectively x. Hence, for a given ⑀ ) 0, there exists a ␦ ) 0 such that f Ž x1 . y f Ž x 2 . -

⑀

Ž 11.49 .

2Ž q y p .

if < x 1 y x 2 < - ␦ , where x 1 and x 2 are points in w p, q x. If k is chosen large enough so that < x iq1 y x i < - ␦ , and hence < xy x i < - ␦ if x i F xF x iq1 , then from Ž11.48. we obtain

Hp f Ž x . sin nx dx F q

2 Mk n

q

⑀

ky1

ÝH 2 Ž q y p . is0 x

x iq1

dx,

i

or

Hp f Ž x . sin nx dx F q

2 Mk n

q

⑀ 2

,

since ky1

ÝH

x iq1

ky1

dxs

xi

is0

Ý Ž x iq1 y x i .

is0

s q y p. Choosing n large enough so that 2 Mkrn- ⑀r2, we finally get

Hp f Ž x . sin nx dx - ⑀ . q

Ž 11.50 .

Formula Ž11.45. follows from Ž11.50., since ⑀ ) 0 is arbitrary. Formula Ž11.46. can be proved in a similar fashion. I Lemma 11.4.2. exists, then

If f Ž x . is piecewise continuous on w0, b x and f X Ž0q.

H n™⬁ 0 lim

b

f Ž x.

sin nx x

dxs

␲ 2

f Ž 0q . .

492

FOURIER SERIES

Proof. We have that

H0 f Ž x . b

sin nx

dxs f Ž 0q .

x

H0

b

q

sin nx

H0

b

x

dx

f Ž x . y f Ž 0q . x

Ž 11.51 .

sin nx dx.

But

H n™⬁ 0

b

lim

sin nx x

H n™⬁ 0

bn

dxs lim s

dx

x

sin x

⬁

H0

sin x

x

␲

dxs

2

Žsee Gillespie, 1959, page 89.. Furthermore, the function Ž1rx .w f Ž x . y f Ž0q.x is piecewise continuous on w0, b x, since f Ž x . is, and limq

f Ž x . y f Ž 0q .

s f X Ž 0q . ,

x

x™0

which exists. Hence, by Lemma 11.4.1,

H n™⬁ 0 lim

b

f Ž x . y f Ž 0q .

sin nx dxs 0.

x

From Ž11.51. we then have

H n™⬁ 0 lim

b

f Ž x.

sin nx x

␲

dxs

f Ž 0q . .

2

I

. Lemma 11.4.3. If f Ž x . is piecewise continuous on w a, b x, and f X Ž xy 0 , . f Ž xq exist at x , ax b, then 0 0 0 X

H n™⬁ a lim

b

f Ž x.

sin n Ž x y x 0 . xy x 0

dxs

␲

q f Ž xy 0 . q f Ž x0 . .

2

Proof. We have that

Ha f Ž x . b

sin n Ž x y x 0 . xy x 0

dxs

Ha

x0

f Ž x.

Hx

b

q s

H0

0

H0

sin n Ž x y x 0 . xy x 0

f Ž x0 y x .

byx 0

dx

x y x0

f Ž x.

x 0ya

q

sin n Ž x y x 0 .

sin nx

f Ž x0 q x .

x

dx

sin nx x

dx

dx.

493

THE FOURIER INTEGRAL

Lemma 11.4.2 applies to each of the above integrals, since the right-hand . and f X Ž xq . derivatives of f Ž x 0 y x . and f Ž x 0 q x . at xs 0 are yf X Ž xy 0 0 , respectively, and both derivatives exist. Furthermore, lim f Ž x 0 y x . s f Ž xy 0 .

x™0 q

and lim f Ž x 0 q x . s f Ž xq 0 ..

x™0 q

It follows that

H n™⬁ a lim

b

f Ž x.

sin n Ž x y x 0 . xy x 0

dxs

␲

q f Ž xy 0 . q f Ž x0 . .

2

I

Proof of Theorem 11.4.1. The function f Ž x . satisfies the conditions of Lemma 11.4.3 on the interval w a, b x. Hence, at any point x, a- x - b, where X . Ž q. f X Ž xy 0 and f x 0 exist,

H ␭™⬁ a

b

lim

f Ž t.

sin ␭ Ž t y x .

␲

dts

tyx

f Ž xy . q f Ž xq . .

2

Ž 11.52 .

Let us now partition the integral Is

⬁

Hy⬁ f Ž t .

sin ␭Ž t y x .

dt

tyx

as Is

Hy⬁ f Ž t . a

⬁

Hb

q

sin ␭Ž t y x .

f Ž t.

tyx

dtq

sin ␭Ž t y x .

Ha

b

f Ž t.

sin ␭ Ž t y x . tyx

Ž 11.53 .

dt.

tyx

dt

From the first integral in Ž11.53. we have

Hy⬁ f Ž t . a

sin ␭Ž t y x .

dt F

tyx

Hy⬁ a

f Ž t. tyx

dt.

Since t F a and a- x, then < t y x < G xy a. Hence,

Hy⬁ a

f Ž t. tyx

dtF

1

H xy a y⬁ a

f Ž t . dt.

Ž 11.54 .

494

FOURIER SERIES

⬁ < Ž .< The integral on the right-hand side of Ž11.54. exists because Hy⬁ f t dt does. Similarly, from the third integral in Ž11.53. we have, if x- b,

⬁

Hb

f Ž t.

sin ␭Ž t y x .

dt F

tyx

F F

⬁

Hb

f Ž t.

dt

tyx 1

⬁

1

⬁

H by x b

f Ž t . dt

H by x y⬁

f Ž t . dt.

Hence, the first and third integrals in Ž11.53. are convergent. It follows that for any ⑀ ) 0, there exists a positive number N such that if a- yN and b) N, then these integrals will each be less than ⑀r3 in absolute value. Furthermore, by Ž11.52., the absolute value of the difference between the second integral in Ž11.53. and the value Ž␲r2.w f Ž xy. q f Ž xq.x can be made less then ⑀r3, if ␭ is chosen large enough. Consequently, the absolute value of the difference between the value of the integral I and Ž␲r2.w f Ž xy. q f Ž xq.x will be less than ⑀ , if ␭ is chosen large enough. Thus, ⬁

H f Ž t. ␭™⬁ y⬁ lim

sin ␭ Ž t y x .

dts

tyx

␲ 2

f Ž xy . q f Ž xq . .

Ž 11.55 .

The expression sinw ␭Ž t y x .xrŽ t y x . in Ž11.55. can be written as sin ␭ Ž t y x . tyx

s

H0 cos ␭

␣ Ž t y x . d␣ .

Formula Ž11.55. can then be expressed as 1 2

f Ž xy . q f Ž xq . s s

1

⬁

H f Ž t . dtH0 cos ␲ ␭™⬁ y⬁ lim

1

␭

⬁

H d ␣Hy⬁ f Ž t . cos ␲ ␭™⬁ 0 lim

␭

␣ Ž t y x . d␣ ␣ Ž t y x . dt. Ž 11.56 .

The change of the order of integration in Ž11.56. is valid because the integrand in Ž11.56. does not exceed < f Ž t .< in absolute value, so that the ⬁ integral Hy⬁ f Ž t .cosw ␣ Ž t y x .x dt converges uniformly for all ␣ Žsee Carslaw, 1930, page 199; Pinkus and Zafrany, 1997, page 187.. From Ž11.56. we finally obtain 1 2

f Ž xy . q f Ž xq . s

1

⬁

½

⬁

H H f Ž t . cos ␲ 0 y⬁

5

␣ Ž t y x . dt d ␣ .

I

APPROXIMATION OF FUNCTIONS BY TRIGONOMETRIC POLYNOMIALS

495

11.5. APPROXIMATION OF FUNCTIONS BY TRIGONOMETRIC POLYNOMIALS By a trigonometric polynomial of the nth order it is meant an expression of the form tn Ž x . s

␣0 2

n

q

Ý w ␣ k cos kxq ␤k sin kx x .

Ž 11.57 .

ks1

A theorem of Weierstrass states that any continuous function of period 2␲ can be uniformly approximated by a trigonometric polynomial of some order Žsee, for example, Tolstov, 1962, Chapter 5.. Thus, for a given ⑀ ) 0, there exists a trigonometric polynomial of the form Ž11.57. such that f Ž x . y tn Ž x . - ⑀ for all values of x. In case the Fourier series for f Ž x . is uniformly convergent, then t nŽ x . can be chosen to be equal to snŽ x ., the nth partial sum of the Fourier series. However, it should be noted that t nŽ x . is not merely a partial sum of the Fourier series for f Ž x ., since a continuous function may have a divergent Fourier series Žsee Jackson,1941, page 26.. We now show that snŽ x . has a certain optimal property among all trigonometric polynomials of the same order. To demonstrate this fact, let f Ž x . be Riemann integrable on wy␲ , ␲ x, and let snŽ x . be the partial sum of order n of its Fourier series, that is, snŽ x . s a0r2 q Ý nks1 w a k cos kxq bk sin kx x. Let rnŽ x . s f Ž x . y snŽ x .. Then, from Ž11.2., ␲

␲

Hy␲ f Ž x . cos kx dxsHy␲ s Ž x . cos kx dx n

s ␲ ak ,

k s 0, 1, . . . , n.

Hence, ␲

Hy␲r Ž x . cos kx dxs 0 n

for k F n.

Ž 11.58 .

for k F n.

Ž 11.59 .

We can similarly show that ␲

Hy␲r Ž x . sin kx dxs 0 n

496

FOURIER SERIES

Now, let u nŽ x . s t nŽ x . y snŽ x ., where t nŽ x . is given by Ž11.57.. Then ␲

Hy␲

f Ž x . y tn Ž x .

2

dxs s s

␲

Hy␲ ␲

Hy␲r

rn Ž x . y u n Ž x . 2 n

␲

Hy␲

2

dx

␲

␲

y␲

y␲

Ž x . dxy 2H rn Ž x . u n Ž x . dxq H u 2n Ž x . dx

f Ž x . y sn Ž x .

2

dxq

␲

Hy␲ u

2 n

Ž x . dx,

Ž 11.60 .

2

Ž 11.61 .

since, by Ž11.58. and Ž11.59., ␲

Hy␲r Ž x . u Ž x . dxs 0. n

n

From Ž11.60. it follows that ␲

Hy␲

f Ž x . y tn Ž x .

2

dxG

␲

Hy␲

f Ž x . y sn Ž x .

dx.

␲ w Ž . This shows that for all trigonometric polynomials of order n, Hy ␲ f x y 2 Ž .x Ž . Ž . t n x dx is minimized when t n x s sn x .

11.5.1. Parseval’s Theorem Suppose that we have the Fourier series Ž11.5. for the function f Ž x ., which is assumed to be continuous of period 2␲ . Let snŽ x . be the nth partial sum of the series. We recall from the proof of Lemma 11.2.1 that ␲

Hy␲

f Ž x . y sn Ž x .

2

dxs

␲

Hy␲ f

2

␲ a20

Ž x . dxy

2

n

q␲

Ý Ž a2k q bk2 .

. Ž 11.62 .

ks1

We also recall that for a given ⑀ ) 0, there exists a trigonometric polynomial t nŽ x . of order n such that f Ž x . y tn Ž x . - ⑀ . Hence, ␲

Hy␲

f Ž x . y tn Ž x .

2

dx- 2␲⑀ 2 .

497

THE FOURIER TRANSFORM

Applying Ž11.61., we obtain ␲

Hy␲

f Ž x . y sn Ž x .

2

dxF

␲

Hy␲

f Ž x . y tn Ž x .

2

dx

Ž 11.63 .

- 2␲⑀ 2 .

Since ⑀ ) 0 is arbitrary, we may conclude from Ž11.62. and Ž11.63. that the limit of the right-hand side of Ž11.62. is zero as n ™ ⬁, that is, 1

␲

H f ␲ y␲

2

Ž x . dxs

a20 2

q

⬁

Ý Ž a2k q bk2 . .

ks1

This result is known as Parse®al’s theorem after Marc Antoine Parseval Ž1755᎐1836..

11.6. THE FOURIER TRANSFORM In the previous sections we discussed Fourier series for functions defined on a finite interval Žor periodic functions defined on R, the set of all real numbers.. In this section, we study a particular transformation of functions defined on R which are not periodic. Let f Ž x . be defined on R s Žy⬁, ⬁.. The Fourier transform of f Ž x . is a function defined on R as FŽ w. s

1

⬁

H f Ž x. e 2␲ y⬁

yi w x

dx,

Ž 11.64 .

where i the complex number 'y 1 , and eyi w x s cos wxy i sin wx. A proper understanding of such a transformation requires some knowledge of complex analysis, which is beyond the scope of this book. However, due to the importance and prevalence of the use of this transformation in various fields of science and engineering, some coverage of its properties is necessary. For this reason, we merely state some basic results and properties concerning this transformation. For more details, the reader is referred to standard books on Fourier series, for example, Pinkus and Zafrany Ž1997, Chapter 3., Kufner and Kadlec Ž1971, Chapter 8., and Weaver Ž1989, Chapter 6.. Theorem 11.6.1. If f Ž x . is absolutely integrable on R, then its Fourier transform F Ž w . exists.

498

FOURIER SERIES

Theorem 11.6.2. If f Ž x . is piecewise continuous and absolutely integrable on R, then its Fourier transform F Ž w . has the following properties: a. F Ž w . is a continuous function on R. b. lim w ™ .⬁ F Ž w . s 0. Note that f Ž x . is piecewise continuous on R if it is piecewise continuous on each finite interval w a, b x. EXAMPLE 11.6.1. on R, since

Let f Ž x . s ey< x < . This function is absolutely integrable ⬁

Hy⬁e

⬁

H0 e

y x

dxs 2

yx

dx

s 2. Its Fourier transform is given by FŽ w. s s s s

1

⬁

1

⬁

1

⬁

H e 2␲ y⬁

y x

H e 2␲ y⬁

y x

H e 2␲ y⬁ 1

⬁

He ␲ 0

y x

yx

eyi w x dx

Ž cos wxy i sin wx . dx cos wx dx

cos wx dx.

Integrating by parts twice, it can be shown that FŽ w. s EXAMPLE 11.6.2.

1

␲ Ž1qw2 .

.

Consider the function f Ž x. s

½

1 0

x F a, otherwise,

where a is a finite positive number. This function is absolutely integrable on R, since ⬁

Hy⬁

f Ž x . dxs

Hya dx a

s 2 a.

499

THE FOURIER TRANSFORM

Its Fourier transform is given by FŽ w. s s s

1

H e 2␲ ya a

1 2␲ iw sin wa

␲w

yi w x

dx

Ž e iw a y eyi w a . .

The next theorem gives the condition that makes it possible to express the function f Ž x . in terms of its Fourier transform using the so-called in®erse Fourier transform. Theorem 11.6.3. Let f Ž x . be piecewise continuous and absolutely integrable on R. Then for every point xg R where f X Ž xy. and f X Ž xq. exist, we have 1 2

f Ž xy . q f Ž xq . s

⬁

Hy⬁ F Ž w . e

iw x

dw.

In particular, if f Ž x . is continuous on R, then f Ž x. s

⬁

Hy⬁ F Ž w . e

iw x

dw.

Ž 11.65 .

By applying Theorem 11.6.3 to the function in Example 11.6.1, we obtain ey

x

e iw x

⬁

s

Hy⬁ ␲ Ž 1 q w

s

Hy⬁ ␲ Ž 1 q w

s

Hy⬁ ␲ Ž 1 q w

s

.

2

.

2

.

1

⬁

cos wx

⬁

2

2

⬁

H ␲ 0

cos wx 1qw2

dw

Ž cos wxq i sin wx . dw dw

dw.

11.6.1. Fourier Transform of a Convolution Let f Ž x . and g Ž x . be absolutely integrable functions on R. By definition, the function hŽ x . s

⬁

Hy⬁ f Ž xy y . g Ž y . dy

Ž 11.66 .

is called the convolution of f Ž x . and g Ž x . and is denoted by Ž f ) g .Ž x ..

500

FOURIER SERIES

Theorem 11.6.4. Let f Ž x . and g Ž x . be absolutely integrable on R. Let F Ž w . and G Ž w . be their respective Fourier transforms. Then, the Fourier transform of the convolution Ž f ) g .Ž x . is given by 2␲ F Ž w .G Ž w ..

11.7. APPLICATIONS IN STATISTICS Fourier series have been used in a wide variety of areas in statistics, such as time series, stochastic processes, approximation of probability distribution functions, and the modeling of a periodic response variable, to name just a few. In addition, the methods and results of Fourier analysis have been effectively utilized in the analytic theory of probability Žsee, for example, Kawata, 1972.. 11.7.1. Applications in Time Series A time series is a collection of observations made sequentially in time. Examples of time series can be found in a variety of fields ranging from economics to engineering. Many types of time series occur in the physical sciences, particularly in meteorology, such as the study of rainfall on successive days, as well as in marine science and geophysics. The stimulus for the use of Fourier methods in time series analysis is the recognition that when observing data over time, some aspects of an observed physical phenomenon tend to exhibit cycles or periodicities. Therefore, when considering a model to represent such data, it is natural to use models that contain sines and cosines, that is, trigonometric models, to describe the behavior. Let y 1 , y 2 , . . . , yn denote a time series consisting of n observations obtained over time. These observations can be represented by the trigonometric polynomial model yt s

a0 2

m

q

Ý w ak cos ␻ k t q bk sin ␻ k t x ,

t s 1, 2, . . . , n,

ks1

where

␻k s ak s bk s

2␲ k n 2 n 2 n

,

k s 0, 1, 2, . . . , m,

n

Ý yt cos ␻ k t ,

k s 0, 1, . . . , m,

ts1 n

Ý yt sin ␻ k t ,

ts1

k s 1, 2, . . . , m.

501

APPLICATIONS IN STATISTICS

The values ␻ 1 , ␻ 2 , . . . , ␻ m are called harmonic frequencies. This model provides a decomposition of the time series into a set of cycles based on the harmonic frequencies. Here, n is assumed to be odd and equal to 2 m q 1, so that the harmonic frequencies lie in the range 0 to ␲ . The expressions for a k Ž k s 0, 1, . . . , m. and bk Ž k s 1, 2, . . . , m. were obtained by treating the model as a linear regression model with 2 m q 1 parameters and then fitting it to the 2 m q 1 observations by the method of least squares. See, for example, Fuller Ž1976, Chapter 7.. The quantity In Ž ␻ k . s

n 2

Ž a2k q bk2 . ,

k s 1, 2, . . . , m,

Ž 11.67 .

represents the sum of squares associated with the frequency ␻ k . For k s 1, 2, . . . , m, the quantities in Ž11.67. define the so-called periodogram. If y 1 , y 2 , . . . , yn are independently distributed as normal variates with zero means and variances ␴ 2 , then the a k ’s and bk ’s, being linear combinations of the yt ’s, will be normally distributed. They are also independent, since the sine and cosine functions are orthogonal. It follows that w nrŽ2 ␴ 2 .xŽ a2k q bk2 ., for k s 1, 2, . . . , m, are distributed as independent chi-squared variates with two degrees of freedom each. The periodogram can be used to search for cycles or periodicities in the data. Much of time series data analysis is based on the Fourier transform and its efficient computation. For more details concerning Fourier analysis of time series, the reader is referred to Bloomfield Ž1976. and Otnes and Enochson Ž1978.. 11.7.2. Representation of Probability Distributions One of the interesting applications of Fourier series in statistics is in providing a representation that can be used to evaluate the distribution function of a random variable with a finite range. Woods and Posten Ž1977. introduced two such representations by combining the concepts of Fourier series and Chebyshev polynomials of the first kind Žsee Section 10.4.1.. These representations are given by the following two theorems: Theorem 11.7.1. Let X be a random variable with a cumulative distribution function F Ž x . defined on w0, 1x. Then, F Ž x . can be represented as a Fourier series of the form

° ¢

0, F Ž x . s 1 y ␪r␲ y Ý⬁ns1 bn sin n␪ , 1,

~

x - 0, 0 F xF 1, x) 1,

502

FOURIER SERIES

where ␪ s ArccosŽ2 xy 1., bn s w2rŽ n␲ .x E w TnU Ž X .x, and E w TnU Ž X .x is the expected value of the random variable TnU Ž X . s cos n Arccos Ž 2 X y 1 . ,

0 F X F 1.

Ž 11.68 .

Note that TnU Ž x . is basically a Chebyshev polynomial of the first kind and of the nth degree defined on w0, 1x. Proof. See Theorem 1 in Woods and Posten Ž1977..

I

The second representation theorem is similar to Theorem 11.7.1, except that X is now assumed to be a random variable over wy1, 1x. Theorem 11.7.2. Let X be a random variable with a cumulative distribution function F Ž x . defined on wy1, 1x. Then

° ¢

0, F Ž x . s 1 y ␪r␲ y Ý⬁ns1 bn sin n␪ , 1,

~

x- y1, y1 F x F 1, x ) 1,

where ␪ s Arccos x, bn s w2rŽ n␲ .x E w TnŽ X .x, E w TnŽ X .x is the expected value of the random variable Tn Ž X . s cos w n Arccos X x ,

y1 F X F 1,

and TnŽ x . is Chebyshev polynomial of the first kind and the nth degree wsee formula Ž10.12.x. Proof. See Theorem 2 in Woods and Posten Ž1977..

I

To evaluate the Fourier series representation of F Ž x ., we must first compute the coefficients bn . For example, in Theorem 11.7.2, bn s w2rŽ n␲ .x E w TnŽ X .x. Since the Chebyshev polynomial TnŽ x . can be written in the form Tn Ž x . s

n

Ý ␣nk x k ,

n s 1, 2, . . . ,

ks0

the computation of bn is equivalent to evaluating bn s

2 n␲

n

Ý ␣ n k ␮Xk ,

n s 1, 2, . . . ,

ks0

where ␮Xk s E Ž X k . is the kth noncentral moment of X. The coefficients ␣ n k can be obtained by using the recurrence relation Ž10.16., that is, Tnq1 Ž x . s 2 xTn Ž x . y Tny1 Ž x . ,

n s 1, 2, . . . ,

503

APPLICATIONS IN STATISTICS

with T0 Ž x . s 1, T1Ž x . s x. This allows us to evaluate the ␣ n k ’s recursively. The series

␪

FŽ x. s1y

␲

y

⬁

Ý bn sin n␪

ns1

is then truncated at n s N. Thus FŽ x. f1y

␪ ␲

N

y

Ý bk sin k␪ .

ks1

Several values of N can be tried to determine the sensitivity of the approximation. We note that this series expansion provides an approximation of F Ž x . in terms of the noncentral moments of X. Good estimates of these moments should therefore be available. It is also possible to extend the applications of Theorems 11.7.1 and 11.7.2 to a random variable X with an infinite range provided that there exists a transformation which transforms X to a random variable Y over w0, 1x, or over wy1, 1x, such that the moments of Y are known from the moments of X. In another application, Fettis Ž1976. developed a Fourier series expansion for Pearson Type IV distributions. These are density functions, f Ž x ., that satisfy the differential equation df Ž x . dx

s

y Ž x q a. c 0 q c1 xq c 2 x 2

f Ž x. ,

where a, c 0 , c1 , and c 2 are constants determined from the central moments ␮ 1 , ␮ 2 , ␮ 3 , and ␮4 , which can be estimated from the raw data. The data are standardized so that ␮ 1 s 0, ␮ 2 s 1. This results in the following expressions for a, c 0 , c1 , and c 2 : c0 s

2␣y1 2 Ž ␣ q 1.

,

c1 s a s

c2 s

␮ 3 Ž ␣ y 1. ␣q1 1 2 Ž ␣ q 1.

,

,

504

FOURIER SERIES

where

␣s

3 Ž ␮4 y ␮23 y 1 . 2 ␮4 y 3 ␮23 y 6

.

Fettis Ž1976. provided additional details that explain how to approximate the cumulative distribution function, FŽ x. s

Hy⬁ f Ž t . dt, x

using Fourier series. 11.7.3. Regression Modeling In regression analysis and response surface methodology, it is quite common to use polynomial models to approximate the mean ␩ of a response variable. There are, however, situations in which polynomial models are not adequate representatives of the mean response, as when ␩ is known to be a periodic function. In this case, it is more appropriate to use an approximating function which is itself periodic. Kupper Ž1972. proposed using partial sums of Fourier series as possible models to approximate the mean response. Consider the following trigonometric polynomial of order d, d

␩ s ␣0 q

Ý w ␣ n cos n␾ q ␤n sin n␾ x ,

Ž 11.69 .

us1

where 0 F ␾ F 2␲ represents either a variable taking values on the real line between 0 and 2␲ , or the angle associated with the polar coordinates of a point on the unit circle. Let u s Ž u1 , u 2 .X , where u1 s cos ␾ , u 2 s sin ␾ . Then, when ds 2, the model in Ž11.69. can be written as

␩ s ␣ 0 q ␣ 1 u1 q ␤ 1 u 2 q ␣ 2 u12 y ␣ 2 u 22 q 2 ␤ 2 u1 u 2 ,

Ž 11.70 .

since sin 2 ␾ s 2 sin ␾ cos ␾ s 2 u1 u 2 , and cos 2 ␾ s cos 2 ␾ y sin 2 ␾ s u12 y u 22 . One of the objectives of response surface methodology is the determination of optimum settings of the model’s control variables that result in a maximum Žor minimum. predicted response. The predicted response ˆ y at a point provides an estimate of ␩ in Ž11.69. and is obtained by replacing ␣ 0 , ␣ n , ␤n in Ž11.69. by their least-squares estimates ␣ ˆ0 , ␣ˆn , and ␤ˆn , respectively, n s 1, 2, . . . , d. For example, if ds 2, we have 2

ˆy s ␣ˆ0 q Ý ␣ˆn cos n ␾ q ␤ˆn sin n ␾ , ns1

Ž 11.71 .

505

APPLICATIONS IN STATISTICS

which can be expressed using Ž11.70. as

ˆ ˆy s ␣ˆ0 q uXˆb q uX Bu, where ˆb s Ž ␣ ˆ1 , ␤ˆ1 .X and

ˆs B

␣ ˆ2

␤ˆ2

␤ˆ2

y␣ ˆ2

with uX u s 1. The method of Lagrange multipliers can then be used to determine the stationary points of ˆ y subject to the constraint uX u s 1. Details of this procedure are given in Kupper Ž1972, Section 3.. In a follow-up paper, Kupper Ž1973. presented some results on the construction of optimal designs for model Ž11.69.. More recently, Anderson-Cook Ž2000. used model Ž11.71. in experimental situations involving cylindrical data. For such data, it is of interest to model the relationship between two correlated components, one a standard linear measurement y, and the other an angular measure ␾ . Examples of such data arise, for example, in biology Žplant or animal migration patterns ., and geology Ždirection and magnitude of magnetic fields.. The fitting of model Ž11.71. is done by using the method of ordinary least squares with the assumption that y is normally distributed and has a constant variance. Anderson-Cook used an example, originally presented in Mardia and Sutton Ž1978., of a cylindrical data set in which y is temperature Žmeasured in degrees Fahrenheit . and ␾ is wind direction Žmeasured in radians .. Based on this example, the fitted model is

ˆy s 41.33 y 2.43 cos ␾ y 2.60 sin ␾ q 3.05 cos 2 ␾ q 2.98 sin 2 ␾ . The corresponding standard errors of ␣ ˆ0 , ␣ˆ1 , ␤ˆ1 , ␣ˆ2 , ␤ˆ2 are 1.1896, 1.6608, 1.7057, 1.4029, 1.7172, respectively. Both ␣ 0 and ␣ 2 are significant parameters at the 5% level, and ␤ 2 is significant at the 10% level. 11.7.4. The Characteristic Function We have seen that the moment generating function ␾ Ž t . for a random variable X is used to obtain the moments of X Žsee Section 5.6.2 and Example 6.9.8.. It may be recalled, however, that ␾ Ž t . may not be defined for all values of t. To generate all the moments of X, it is sufficient for ␾ Ž t . to be defined in a neighborhood of t s 0 Žsee Section 5.6.2.. Some well-known distributions do not have moment generating functions, such as the Cauchy distribution Žsee Example 6.9.1.. Another function that generates the moments of a random variable in a manner similar to ␾ Ž t ., but is defined for all values of t and for all random

506

FOURIER SERIES

variables, is the characteristic function. By definition, the characteristic function of a random variable X, denoted by ␾c Ž t ., is

␾c Ž t . s E w e it X x s

⬁

Hy⬁e

it x

dF Ž x . ,

Ž 11.72 .

where F Ž x . is the cumulative distribution function of X, and i is the complex number 'y 1 . If X is discrete and has the values c1 , c 2 , . . . , c n , . . . , then Ž11.72. takes the form

␾c Ž t . s

⬁

Ý p Ž c j . e it c , j

Ž 11.73 .

js1

where pŽ c j . s P w X s c j x, j s 1, 2, . . . . If X is continuous with the density function f Ž x ., then

␾c Ž t . s

⬁

Hy⬁e

it x

f Ž x . dx.

Ž 11.74 .

The function ␾c Ž t . is complex-valued in general, but is defined for all values ⬁ ⬁ of t, since e it x s cos txq i sin tx, and both Hy⬁ cos tx dF Ž x . and Hy⬁ sin tx dF Ž x . exist by the fact that ⬁

Hy⬁ cos tx ⬁

Hy⬁ sin tx

⬁

dF Ž x . F

Hy⬁ dF Ž x . s 1,

dF Ž x . F

Hy⬁ dF Ž x . s 1.

⬁

The characteristic function and the moment generating function, when the latter exists, are related according to the formula

␾c Ž t . s ␾ Ž it . . Furthermore, it can be shown that if X has finite moments, then they can be obtained by repeatedly differentiating ␾c Ž t . and evaluating the derivatives at zero, that is, EŽ X n. s

1 d n␾c Ž t . in

dt n

,

n s 1, 2, . . . .

ts0

Although ␾c Ž t . generates moments, it is mainly used as a tool to derive distributions. For example, from Ž11.74. we note that when X is continuous, the characteristic function is a Fourier-type transformation of the density

507

APPLICATIONS IN STATISTICS

function f Ž x .. This follows from Ž11.64., the Fourier transform of f Ž x ., which ⬁ f Ž x . eyi t x dx. If we denote this transform by G Ž t . , is given by Ž1r2␲ .Hy⬁ then the relationship between ␾c Ž t . and GŽ t . is given by

␾c Ž t . s 2␲ G Ž yt . . By Theorem 11.6.3, if f Ž x . is continuous and absolutely integrable on R, then f Ž x . can be derived from ␾c Ž t . by using formula Ž11.65., which can be written as f Ž x. s s s

⬁

Hy⬁G Ž t . e 1

it x

dt

⬁

H ␾ Ž yt . e 2␲ y⬁

it x

c

1

⬁

H ␾ Ž t. e 2␲ y⬁

yi t x

c

dt

dt.

Ž 11.75 .

This is known as the in®ersion formula for characteristic functions. Thus the distribution of X can be uniquely determined by its characteristic function. There is therefore a one-to-one correspondence between distribution functions and their corresponding characteristic functions. This provides a useful tool for deriving distributions of random variables that cannot be easily calculated, but whose characteristic functions are straightforward. Waller, Turnbull, and Hardin Ž1995. reviewed and discussed several algorithms for inverting characteristic functions, and gave several examples from various areas in statistics. Waller Ž1995. demonstrated that characteristic functions provide information beyond what is given by moment generating functions. He pointed out that moment generating functions may be of more mathematical than numerical use in characterizing distributions. He used an example to illustrate that numerical techniques using characteristic functions can differentiate between two distributions, even though their moment generating functions are very similar Žsee also McCullagh, 1994.. Luceno ˜ Ž1997. provided further and more general arguments to show that characteristic functions are superior to moment generating and probability generating functions Žsee Section 5.6.2. in their numerical behavior. One of the principal uses of characteristic functions is in deriving limiting distributions. This is based on the following theorem Žsee, for example, Pfeiffer, 1990, page 426.: Theorem 11.7.3. Consider the sequence
FnŽ x .4⬁ns1 of cumulative distribution functions. Let
␾nc Ž t .4⬁ns1 be the corresponding sequence of characteristic functions. a. If FnŽ x . converges to a distribution function F Ž x . at every point of continuity for F Ž x ., then ␾nc Ž t . converges to ␾c Ž t . for all t, where ␾c Ž t . is the characteristic function for F Ž x ..

508

FOURIER SERIES

b. If ␾nc Ž t . converges to ␾c Ž t . for all t and ␾c Ž t . is continuous at t s 0, then ␾c Ž t . is the characteristic function for a distribution function F Ž x . such that FnŽ x . converges to F Ž x . at each point of continuity of F Ž x .. It should be noted that in Theorem 11.7.3 the condition that the limiting function ␾c Ž t . is continuous at t s 0 is essential for the validity of the theorem. The following example shows that if this condition is violated, then the theorem is no longer true: Consider the cumulative distribution function

°0,xq n

xF yn,

~

Fn Ž x . s

yn- x- n,

,

¢1,2 n

xG n.

The corresponding characteristic function is

␾nc Ž t . s s

1

H e 2 n yn n

sin nt nt

it x

dx

.

As n ™ ⬁, ␾nc Ž t . converges for every t to ␾c Ž t . defined by

␾c Ž t . s

½

t s 0, t / 0.

1, 0,

Thus, ␾c Ž t . is not continuous for t s 0. We note, however, that FnŽ x . ™ 12 for every fixed x. Hence, the limit of FnŽ x . is not a cumulative distribution function. EXAMPLE 11.7.1. Consider the distribution defined by the density function f Ž x . s eyx for x) 0. Its characteristic function is given by

H0 e

s

H0 e

s EXAMPLE 11.7.2.

⬁

␾c Ž t . s

⬁

i t x yx

e

dx

yx Ž1yi t .

1 1 y it

dx

.

Consider the Cauchy density function f Ž x. s

1

␲ Ž1qx2 .

,

y⬁ - x- ⬁,

509

APPLICATIONS IN STATISTICS

given in Example 6.9.1. The characteristic function is

␾c Ž t . s s

s

1

␲ 1

e it x

⬁

Hy⬁ 1 q x

cos tx

⬁

H ␲ y⬁ 1 q x 1

2

2

cos tx

⬁

H ␲ y⬁ 1 q x

2

dx

dxq

i

sin tx

⬁

H ␲ y⬁

1qx2

dx

dx

s ey < t < . Note that this function is not differentiable at t s 0. We may recall from Example 6.9.1 that all moments of the Cauchy distribution do not exist. EXAMPLE 11.7.3. In Example 6.9.8 we saw that the moment generating function for the gamma distribution G Ž ␣ , ␤ . with the density function f Ž x. s

x ␣y1 eyx r ␤

␣ ) 0,

,

⌫Ž ␣ . ␤ ␣

␤ ) 0, 0 - x- ⬁,

is ␾ Ž t . s Ž1 y ␤ t .y␣ . Hence, its characteristic function is ␾c Ž t . s Ž1 y i ␤ t .y␣ . EXAMPLE 11.7.4. The characteristic function of the standard normal distribution with the density function f Ž x. s

1

'2␲

eyx

2

r2

y⬁ - x - ⬁,

,

is

␾c Ž t . s s

s

1

⬁

1

⬁

'2␲ Hy⬁e

yx 2 r2 it x

e

dx

'2␲ Hy⬁e eyt

2

r2

'2␲

s eyt

2

r2

.

y 12 Ž x 2 y2 it x .

⬁

Hy⬁e

y 12 Ž xyi t . 2

dx

dx

510

FOURIER SERIES

Vice versa, the density function can be retrieved from ␾c Ž t . by using the inversion formula Ž11.75.: f Ž x. s s s

s

s

s

1

⬁

1

⬁

1

⬁

H e 2␲ y⬁

yt 2 r2 yi t x

e

H e 2␲ y⬁

y 12 Ž t 2 q2 it x .

dt dt

H e 2␲ y⬁ eyx

2

eyx

2

eyx

2

r2

'2␲ r2

'2␲

y 12 w t 2 q2 t Ž i x .q Ž i x . 2 x Ž i x . 2 r2

e

⬁

1

⬁

1

Hy⬁ '2␲ e

y 12 Ž tqi x . 2

Hy⬁ '2␲ e

yu 2 r2

dt

dt

du

r2

'2␲

.

11.7.4.1. Some Properties of Characteristic Functions The book by Lukacs Ž1970. provides a detailed study of characteristic functions and their properties. Proofs of the following theorems can be found in Chapter 2 of that book. Theorem 11.7.4. Every characteristic function is uniformly continuous on the whole real line. Theorem 11.7.5. Suppose that ␾ 1 c Ž t ., ␾ 2 c Ž t ., . . . , ␾nc Ž t . are characteristic functions. Let a1 , a2 , . . . , a n be nonnegative numbers such that Ý nis1 a i s 1. Then Ý nis1 a i ␾ ic Ž t . is also a characteristic function. Theorem 11.7.6. The characteristic function of the convolution of two distribution functions is the product of their characteristic functions. Theorem 11.7.7. teristic function.

The product of two characteristic functions is a charac-

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Anderson-Cook, C. M. Ž2000.. ‘‘A second order model for cylindrical data.’’ J. Statist. Comput. Simul., 66, 51᎐65.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

511

Bloomfield, P. Ž1976.. Fourier Analysis of Time Series: An Introduction. Wiley, New York. ŽThis is an introductory text on Fourier methods written at an applied level for users of time series. . Carslaw, H. S. Ž1930.. Introduction to the Theory of Fourier Series and Integrals, 3rd ed. Dover, New York. Churchill, R. V. Ž1963.. Fourier Series and Boundary Value Problems, 2nd ed. McGraw-Hill, New York. ŽThis text provides an introductory treatment of Fourier series and their applications to boundary value problems in partial differential equations of engineering and physics. Fourier integral representations and expansions in series of Bessel functions and Legendre polynomials are also treated. . Davis, H. F. Ž1963.. Fourier Series and Orthogonal Functions. Allyn & Bacon, Boston. Fettis, H. E. Ž1976.. ‘‘Fourier series expansions for Pearson Type IV distributions and probabilities.’’ SIAM J. Applied Math., 31, 511᎐518. Fuller, W. A. Ž1976.. Introduction to Statistical Time Series. Wiley, New York. Gillespie, R. P. Ž1959.. Integration. Oliver and Boyd, London. Jackson, D. Ž1941.. Fourier Series and Orthogonal Polynomials. Mathematical Association of America, Washington. Kawata, T. Ž1972.. Fourier Analysis in Probability Theory. Academic Press, New York. ŽThis text presents useful results from the theories of Fourier series, Fourier transforms, Laplace transforms, and other related topics that are pertinent to the study of probability theory.. Kufner, A., and J. Kadlec Ž1971.. Fourier Series. Iliffe BooksᎏThe Butterworth Group, London. ŽThis is an English translation edited by G. A. Toombs.. Kupper, L. L. Ž1972.. ‘‘Fourier series and spherical harmonics regression.’’ Appl. Statist., 21, 121᎐130. Kupper, L. L. Ž1973.. ‘‘Minimax designs for Fourier series and spherical harmonics regressions: A characterization of rotatable arrangements.’’ J. Roy. Statist. Soc., Ser. B, 35, 493᎐500. Luceno, ˜ A. Ž1997.. ‘‘Further evidence supporting the numerical usefulness of characteristic functions.’’ Amer. Statist., 51, 233᎐234. Lukacs, E. Ž1970.. Characteristic Functions, 2nd ed. Hafner, New York. ŽThis is a classic book covering many interesting details concerning characteristic functions. . Mardia, K. V., and T. W. Sutton Ž1978.. ‘‘Model for cylindrical variables with applications.’’ J. Roy. Statist. Soc., Ser. B, 40, 229᎐233. McCullagh, P. Ž1994.. ‘‘Does the moment-generating function characterize a distribution?’’ Amer. Statist., 48, 208. Otnes, R. K., and L. Enochson Ž1978.. Applied Time Series Analysis. Wiley, New York. Pfeiffer, P. E. Ž1990.. Probability for Applications. Springer-Verlag, New York. Pinkus, A., and S. Zafrany Ž1997.. Fourier Series and Integral Transforms. Cambridge University Press, Cambridge, England. Tolstov, G. P. Ž1962.. Fourier Series. Dover, New York. ŽTranslated from the Russian by Richard A. Silverman.. Waller, L. A. Ž1995.. ‘‘Does the characteristic function numerically distinguish distributions?’’ Amer. Statist., 49, 150᎐152.

512

FOURIER SERIES

Waller, L. A., B. W. Turnbull, and J. M. Hardin Ž1995.. ‘‘Obtaining distribution functions by numerical inversion of characteristic functions with applications.’’ Amer. Statist., 49, 346᎐350. Wea®er, H. J. Ž1989.. Theory of Discrete and Continuous Fourier Analysis. Wiley, New York. Woods, J. D., and H. O. Posten Ž1977.. ‘‘The use of Fourier series in the evaluation of probability distribution functions.’’ Commun. Statist.ᎏSimul. Comput., 6, 201᎐219.

EXERCISES In Mathematics 11.1. Expand the following functions using Fourier series: (a) f Ž x . s < x < , y␲ F xF ␲ . (b) f Ž x . s < sin x < . (c) f Ž x . s xq x 2 , y␲ F xF ␲ . 11.2. Show that ⬁

1

ns1

Ž 2 n y 1.

Ý

2

s

␲2 8

.

w Hint: Use the Fourier series for x 2 .x 11.3. Let a n and bn be the Fourier coefficients for a continuous function f Ž x . defined on wy␲ , ␲ x such that f Žy␲ . s f Ž␲ ., and f X Ž x . is piecewise continuous on wy␲ , ␲ x. Show that (a) lim n™⬁Ž nan . s 0, (b) lim n™⬁Ž nbn . s 0. 11.4. If f Ž x . is continuous on wy␲ , ␲ x, f Žy␲ . s f Ž␲ ., and f X Ž x . is piecewise continuous on wy␲ , ␲ x, then show that f Ž x . y sn Ž x . F

c

'n

,

where sn Ž x . s

a0

n

q

2

Ý w ak cos kxq bk sin kx x ,

ks1

and c2 s

1

␲

H f ␲ y␲

X2

Ž x . dx.

513

EXERCISES

11.5. Suppose that f Ž x . is piecewise continuous on wy␲ , ␲ x and has the Fourier series given in Ž11.32.. (a) Show that Ý⬁ns1 Žy1. n bnrn is a convergent series. (b) Show that Ý⬁ns1 bnrn is convergent. w Hint: Use Theorem 11.3.2.x 11.6. Show that the trigonometric series, Ý⬁ns2 Žsin nx .rlog n, is not a Fourier series of any integrable function. w Hint: If it were a Fourier series of a function f Ž x ., then bn s 1rlog n would be the Fourier coefficient of an odd function. Apply now part Žb. of Exercise 11.5 and show that this assumption leads to a contradiction.x 11.7. Consider the Fourier series of f Ž x . s x given in Example 11.2.1. (a) Show that x2 4

s

␲2

y

12

⬁

Ý

Ž y1.

nq1

n

ns1

cos nx

for y␲ - x- ␲ .

2

w Hint: Consider the Fourier series of H0x f Ž t . dt.x (b) Deduce that ⬁

Ý

Ž y1.

ns1

nq1

s

n2

␲2 12

.

11.8. Make use of the result in Exercise 11.7 to find the sum of the series Ý⬁ns1 wŽy1. nq1 sin nx xrn3. 11.9. Show that the Fourier transform of f Ž x . s eyx , y⬁ - x- ⬁, is given by 2

FŽ w. s

1 2'␲

eyw

2

r4

.

w Hint: Show that F X Ž w . q 12 wF Ž w . s 0.x 11.10. Prove Theorem 11.6.4 using Fubini’s theorem. 11.11. Use the Fourier transform to solve the integral equation ⬁

Hy⬁ f Ž xy y . f Ž y . dy s e for the function f Ž x ..

yx 2 r2

514

FOURIER SERIES

11.12. Consider the function f Ž x . s x, y␲ - x- ␲ , with f Žy␲ . s f Ž␲ . s 0, and f Ž x . 2␲-periodic defined on Žy⬁, ⬁.. The Fourier series of f Ž x . is ⬁

Ý

2 Ž y1 .

nq1

sin nx

n

ns1

.

Let snŽ x . be the nth partial sum of this series, that is, sn Ž x . s

n

Ý

2 Ž y1 .

kq1

sin kx.

k

ks1

Let x n s ␲ y ␲rn. (a) Show that sn Ž x n . s

n

Ý

2 sin Ž k␲rn . k

ks1

.

(b) Show that

n™⬁

␲

H0

lim sn Ž x n . s 2

sin x x

dx

f 1.18␲ . Note: As n ™ ⬁, x n ™ ␲y. Hence, for n sufficiently large, sn Ž x n . y f Ž x n . f 1.18␲ y ␲ s 0.18␲ . Thus, near xs ␲ wa point of discontinuity for f Ž x .x, the partial sums of the Fourier series exceed the value of this function by approximately the amount 0.18␲ s 0.565. This illustrates the socalled Gibbs phenomenon according to which the Fourier series of f Ž x . ‘‘overshoots’’ the value of f Ž x . in a small neighborhood to the left of the point of discontinuity of f Ž x .. It can also be shown that in a small neighborhood to the right of x s y␲ , the Fourier series of f Ž x . ‘‘undershoots’’ the value of f Ž x .. In Statistics 11.13. In the following table, two observations of the resistance in ohms are recorded at each of six equally spaced locations on the perimeter of a

515

EXERCISES

new type of solid circular coil Žsee Kupper, 1972, Table 1.:

␾ Žradians .

Resistance Žohms.

0 ␲r3 2␲r3 ␲ 4␲r3 5␲r3

13.62, 14.40 10.552, 10.602 2.196, 3.696 6.39, 7.25 8.854, 10.684 5.408, 8.488

(a) Use the method of least squares to estimate the parameters in the following trigonometric polynomial of order 2: 2

␩ s ␣0 q

Ý w ␣ n cos n␾ q ␤n sin n␾ x ,

ns1

where 0 F ␾ F 2␲ , and ␩ denotes the average resistance at location ␾ . (b) Use the prediction equation obtained in part Ža. to determine the points of minimum and maximum resistance on the perimeter of the circular coil. 11.14. Consider the following circular data set in which ␾ is wind direction and y is temperature Žsee Anderson᎐Cook, 2000, Table 1..

␾ Žradians .

y Ž⬚F.

␾ Žradians .

y Ž⬚F.

4.36 3.67 4.36 1.57 3.67 3.67 6.11 5.93 0.52 3.67 3.67 3.32 4.89 3.14

52 41 41 31 53 47 43 43 41 46 48 52 43 46

4.54 2.62 2.97 4.01 4.19 5.59 5.59 3.32 3.67 1.22 4.54 4.19 3.49 4.71

38 40 49 48 37 37 33 47 51 42 53 46 51 39

Fit a second y order trigonometric polynomial to this data set, and verify that the prediction equation is given by

ˆy s 41.33 y 2.43 cos ␾ y 2.60 sin ␾ q 3.05 cos 2 ␾ q 2.98 sin 2 ␾ .

516

FOURIER SERIES

11.15. Let
Yn4⬁ns1 be a sequence of independent, indentically distributed random variables with mean ␮ and variance ␴ 2 . Let sUn s

Yn y ␮

␴r'n

,

where Yn s Ž1rn.Ý nis1 Yi . (a) Find the characteristic function of sUn . (b) Use Theorem 11.7.3 and part Ža. to show that the limiting distribution of sUn as n ™ ⬁ is the standard normal distribution. Note: Part Žb. represents the statement of the well-known central limit theorem, which asserts that for large n, the arithmetic mean Yn of a sample of independent, identically distributed random variables is approximately normally distributed with mean ␮ and standard deviation ␴r 'n .

CHAPTER 12

Approximation of Integrals

Integration plays an important role in many fields of science and engineering. For applications, numerical values of integrals are often required. However, in many cases, the evaluation of integrals, or quadrature, by elementary functions may not be feasible. Hence, approximating the value of an integral in a reliable fashion is a problem of utmost importance. Numerical quadrature is in fact one of the oldest branches of mathematics: the determination, approximately or exactly, of the areas of regions bounded by lines or curves, a subject which was studied by the ancient Babylonians Žsee Haber, 1970.. The word ‘‘quadrature’’ indicates the process of measuring an area inside a curve by finding a square having the same area. Probably no other problem has exercised a greater or a longer attraction than that of constructing a square equal in area to a given circle. Thousands of people have worked on this problem, including the ancient Egyptians as far back as 1800 B.C. In this chapter, we provide an exposition of methods for approximating integrals, including those that are multidimensional.

12.1. THE TRAPEZOIDAL METHOD This is the simplest method of approximating an integral of the form Hab f Ž x . dx, which represents the area bounded by the curve of the function y s f Ž x . and the two lines x s a, xs b. The method is based on approximating the curve by a series of straight line segments. As a result, the area is approximated with a series of trapezoids. For this purpose, the interval from a to b is divided into n equal parts by the partition points a s x 0 , x 1 , x 2 , . . . , x n s b. For the ith trapezoid, which lies between x iy1 and x i , its width is h s Ž1rn.Ž by a. and its area is given by

Ai s

h 2

f Ž x iy1 . q f Ž x i . ,

i s 1, 2, . . . , n.

Ž 12.1 . 517

518

APPROXIMATION OF INTEGRALS

The sum, Sn , of A1 , A 2 , . . . , A n provides an approximation to the integral Hab f Ž x . dx. n

Sn s

Ý Ai

is1

s

h 2

s

h 2


f Ž x 0 . q f Ž x1 .

q f Ž x 1 . q f Ž x 2 . q ⭈⭈⭈ q f Ž x ny1 . q f Ž x n .

f Ž x0 . q f Ž xn . q 2

ny1

Ý f Ž xi .

4

Ž 12.2 .

.

is1

12.1.1. Accuracy of the Approximation The accuracy in the trapezoidal method depends on the number n of trapezoids we take. The next theorem provides information concerning the error or approximation. Theorem 12.1.1. Suppose that f Ž x . has a continuous second derivative on w a, b x, and f Y Ž x . F M2 for all x in w a, b x. Then

Ha f Ž x . dxy S b

n

F

3 Ž by a. M2

12 n2

,

where Sn is given by formula Ž12.2.. Proof. Consider the partition points as x 0 , x 1 , x 2 , . . . , x n s b such that h s x i y x iy1 s Ž1rn.Ž by a., i s 1, 2, . . . , n. The integral of f Ž x . from x iy1 to x i is Ii s

Hx

xi

f Ž x . dx.

Ž 12.3 .

iy1

Now, in the trapezoidal method, f Ž x . is approximated in the interval w x iy1 , x i x by the right-hand side of the straight-line equation, pi Ž x . s f Ž x iy1 . q xi y x

1 h

f Ž x i . y f Ž x iy1 . Ž x y x iy1 .

f Ž x iy1 . q

xy x iy1

f Ž xi . h x y x iy1 s f Ž x iy1 . q f Ž xi . , x iy1 y x i x i y x iy1 s

h xy x i

i s 1, 2, . . . , n.

Note that pi Ž x . is a linear Lagrange interpolating polynomial Žof degree n s 1. with x iy1 and x i as its points of interpolation wsee formula Ž9.14.x.

519

THE TRAPEZOIDAL METHOD

Using Theorem 9.2.2, the error of interpolation resulting from approximating f Ž x . with pi Ž x . over w x iy1 , x i x is given by 1

f Ž x . y pi Ž x . s

2!

f Y Ž ␰ i . Ž xy x iy1 . Ž xy x i . ,

i s 1, 2, . . . , n, Ž 12.4 .

where x iy1 - ␰ i - x i . Formula Ž12.4. results from applying formula Ž9.15.. Hence, the error of approximating Ii with A i in Ž12.1. is Ii y A i s s

Hx

xi

1 2!

s

1 2!

s

h 2!

s

f Ž x . y pi Ž x . dx

iy1

h 2!

sy

f Y Ž ␰i .

Hx

xi

Ž xy x iy1 . Ž xy x i . dx

iy1

f Y Ž ␰i .

1 3

3 2 Ž x i3 y x iy1 . y 12 Ž x iy1 q x i . Ž x i2 y x iy1 . q x iy1 x i Ž x i y x iy1 .

f Y Ž ␰i .

1 3

2 Ž x i2 q x iy1 x i q x iy1 . y 12 Ž x iy1 q x i . 2 q x iy1 x i

f Y Ž ␰i .

1 6

2 y x i2 . Ž 2 x iy1 x i y x iy1

h3 12

f Y Ž ␰i . ,

Ž 12.5 .

i s 1, 2, . . . , n.

The total error of approximating Hab f Ž x . dx with Sn is then given by

Ha f Ž x . dxy S b

nsy

h3 12

n

Ý f Y Ž ␰i . .

is1

It follows that

Ha

b

f Ž x . dxy S n F s

nh3 M2 12 3 Ž by a. M2

12 n2

.

I

Ž 12.6 .

An alternative procedure to approximating the integral Hab f Ž x . dx by a sum i of trapezoids is to approximate Hxxiy1 f Ž x . dx by a trapezoid bounded from above by the tangent to the curve of y s f Ž x . at the point x iy1 q hr2, which is the midpoint of the interval w x iy1 , x i x. In this case, the area of the ith trapezoid is

ž

AUi s hf x iy1 q

h 2

/

,

i s 1, 2, . . . , n.

520

APPROXIMATION OF INTEGRALS

Hence,

Ha

b

ž

n

f Ž x . dxf h Ý f x iy1 q is1

h 2

/

Ž 12.7 .

,

and f Ž x . is approximated in the interval w x iy1 , x i x by

ž

pUi Ž x . s f x iy1 q

h 2

/ ž

h

q x y x iy1 y

2

/ ž

f X x iy1 q

h 2

/

.

By applying Taylor’s theorem ŽTheorem 4.3.1. to f Ž x . in a neighborhood of x iy1 q hr2, we obtain

ž

f Ž x . s f x iy1 q q

1 2!

ž

h 2

/ ž

q xy x iy1 y

xy x iy1 y

s pUi Ž x . q

1 2!

ž

h 2

/

h 2

/ ž

f X x iy1 q

h 2

/

2

f Y Ž ␩i . h

xy x iy1 y

2

/

2

f Y Ž ␩i . ,

where ␩i lies between x iy1 q hr2 and x. The error of approximating i Hxxiy1 f Ž x . dx with AUi is then given by

Hx

xi

f Ž x . y pUi Ž x . dxs

iy1

F

1

H 2! x M2 2!

s

xi iy1

Hx

h 3 M2 24

ž

xi iy1

xy x iy1 y

ž

h 2

xy x iy1 y

/

h 2

2

f Y Ž ␩i . dx

/

2

dx

.

Consequently, the absolute value of the total error in this case has an upper bound of the form n

xi

is1

x iy1

ÝH

f Ž x.

y pUi

Ž x . dx F s

nh3 M2 24 3 Ž by a. M2

24 n2

.

Ž 12.8 .

SIMPSON’S METHOD

521

We note that the upper bound in Ž12.8. is half as large as the one in Ž12.6.. This alternative procedure is therefore slightly more precise than the original one. Both procedures produce an approximating error that is O Ž1rn2 .. It should be noted that this error does not include the roundoff errors in the computation of the areas of the approximating trapezoids. EXAMPLE 12.1.1. Consider approximating the integral H12 dxrx, which has an exact value equal to log 2 f 0.693147. Let us divide the interval w1, 2x into n s 10 subintervals of length h s 101 . Hence, x 0 s 1, x 1 s 1.1, . . . , x 10 s 2.0. Using the first trapezoidal method Žformula 12.2., we obtain

H1

2

dx x

f

1 20

1q

1 2

9

1

is1

xi

q2 Ý

s 0.69377. Using now the second trapezoidal method wformula Ž12.7.x, we get

H1

2

dx x

f

1 10

10

Ý

is1

1 x iy1 q 0.05

s 0.69284. 12.2. SIMPSON’S METHOD Let us again consider the integral Hab f Ž x . dx. Let as x 0 - x 1 - ⭈⭈⭈ - x 2 ny1 x 2 n s b be a sequence of equally spaced points that partition the interval w a, b x such that x i y x iy1 s h, i s 1, 2, . . . , 2 n. Simpson’s method is based on approximating the graph of the function f Ž x . over the interval w x iy1 , x iq1 x by a parabola which agrees with f Ž x . at the points x iy1 , x i , and x iq1 . Thus, over w x iy1 , x iq1 x, f Ž x . is approximated with a Lagrange interpolating polynomial of degree 2 of the form wsee formula Ž9.14.x qi Ž x . s f Ž x iy1 . q f Ž xi . s f Ž x iy1 .

Ž xy x i . Ž xy x iq1 . Ž x iy1 y x i . Ž x iy1 y x iq1 . Ž xy x iy1 . Ž xy x iq1 . Ž xy x iy1 . Ž xy x i . q f Ž x iq1 . Ž x i y x iy1 . Ž x i y x iq1 . Ž x iq1 y x iy1 . Ž x iq1 y x i . Ž xy x i . Ž x y x iq1 .

q f Ž x iq1 .

2h

2

y f Ž xi .

Ž xy x iy1 . Ž xy x i . 2 h2

.

Ž xy x iy1 . Ž xy x iq1 . h2

522

APPROXIMATION OF INTEGRALS

It follows that

Hx

x iq1

f Ž x . dxf

iy1

s

Hx

x iq1

qi Ž x . dx

iy1

f Ž x iy1 . 2h y

2

f Ž xi . h

q

2

s

2

f Ž x iy1 . 2 h2 h 3

x iq1

Ž xy x i . Ž xy x iq1 . dx

iy1

Hx

f Ž x iq1 . 2h

s

Hx

x iq1

Ž xy x iy1 . Ž xy x iq1 . dx

iy1

Hx

x iq1

Ž xy x iy1 . Ž xy x i . dx

iy1

ž / 2 h3

y

3

f Ž xi . h2

ž

y4h3 3

f Ž x iy1 . q 4 f Ž x i . q f Ž x iq1 . ,

/

q

f Ž x iq1 . 2 h2

ž / 2 h3 3

i s 1,3, . . . , 2 n y 1.

Ž 12.9 . By adding up all the approximations in Ž12.9. for i s 1, 3, . . . , 2 n y 1, we obtain

Ha

b

f Ž x . dxf

h 3

n

ny1

is1

is1

f Ž x 0 . q 4 Ý f Ž x 2 iy1 . q 2

Ý f Ž x2 i . q f Ž x2 n .

. Ž 12.10 .

As before, in the case of the trapezoidal method, the accuracy of the approximation in Ž12.10. can be figured out by using formula Ž9.15.. Courant and John Ž1965, page 487., however, stated that the error of approximation can be improved by one order of magnitude by using a cubic interpolating polynomial which agrees with f Ž x . at x iy1 , x i , x iq1 , and whose derivative at x i is equal to f X Ž x i .. Such a polynomial gives a better approximation to f Ž x . over w x iy1 , x iq1 xthan the quadratic one, and still provides the same approximation formula Ž12.l0. for the integral. If qi Ž x . is chosen as such, then the error of interpolation resulting from approximating f Ž x . with qi Ž x . over w x iy1 , x iq1 x is given by f Ž x . y qi Ž x . s Ž1r4!. f Ž4. Ž ␰ i .Ž xy x iy1 .Ž x y x i . 2 Ž x y x iq1 ., where x iy1 - ␰ i - x iq1 , provided that f Ž4. Ž x . exists and is continuous on w a, b x. This is equivalent to using formula Ž9.15. with n s 3 and with two of the interpolation points coincident at x i . We then have

Hx

x iq1 iy1

f Ž x . y qi Ž x . dx F

M4

H 4! x

x iq1

2 Ž xy x iy1 . Ž x y x i . Ž x y x iq1 . dx,

iy1

i s 1,3, . . . , 2 n y 1,

Ž 12.11 .

NEWTON᎐ COTES METHOD

523

where M4 is an upper bound on < f Ž4. Ž x .< for aF xF b. By computing the integral in Ž12.11. we obtain

Hx

x iq1

f Ž x . y qi Ž x . dx F

M4 h 5 90

iy1

i s 1, 3, . . . , 2 n y 1.

,

Consequently, the total error of approximation in Ž12.10. is less than or equal to nM4 h5

s

M4 Ž b y a .

5

Ž 12.12 .

,

2880n4

90

since h s Ž by a.r2 n. Thus the error of approximation with Simpson’s method is O Ž1rn4 ., where n is half the number of subintervals into which w a, b x is divided. Hence, Simpson’s method yields a much more accurate approximation than the trapezoidal method. As an example, let us apply Simpson’s method to the calculation of the integral in Example 12.1.1 using the same division of w1, 2x into 10 subintervals, each of length h s 101 . By applying formula Ž12.10. we obtain

H1

2

dx x

f

0.10 3

1q4

ž

1

q

1.1

q2

ž

1 1.2

1

q

1.3 q

1 1.4

1

q

1.5 q

1 1.6

1

q

1.7 q

1 1.8

1 1.9

/

q

/ 1 2

s 0.69315.

12.3. NEWTON–COTES METHODS The trapezoidal and Simpson’s methods are two special cases of a general series of approximate integration methods of the so-called Newton᎐Cotes type. In the trapezoidal method, straight line segments were used to approximate the graph of the function f Ž x . between a and b. In Simpson’s method, the approximation was carried out using a series of parabolas. We can refine this approximation even further by considering a series of cubic curves, quartic curves, and so on. For cubic approximations, four equally spaced points are used to subdivide each subinterval of w a, b x Žinstead of two points for the trapezoidal method and three points for Simpson’s method., whereas five points are needed for quartic approximation, and so on. All such approximations are of the Newton᎐Cotes type.

524

APPROXIMATION OF INTEGRALS

12.4. GAUSSIAN QUADRATURE All Newton᎐Cotes methods require the use of equally spaced points, as was seen in the cases of the trapezoidal method and Simpson’s method. If this requirement is waived, then it is possible to select the points in a manner that reduces the approximation error. Let x 0 - x 1 - x 2 - ⭈⭈⭈ - x n be n q 1 distinct points in w a, b x. Consider the approximation n

Ha f Ž x . dxf Ý ␻ f Ž x . , b

i

i

Ž 12.13 .

is0

where the coefficients, ␻ 0 , ␻ 1 , . . . , ␻ n , are to be determined along with the points x 0 , x 1 , . . . , x n . The total number of unknown quantities in Ž12.13. is 2 n q 2. Hence, 2 n q 2 conditions must be specified. According to the so-called Gaussian integration rule, the ␻ i ’s and x i ’s are chosen such that the approximation in Ž12.13. will be exact for all polynomials of degrees not exceeding 2 n q 1. This is equivalent to requiring that the approximation Ž12.13. be exact for f Ž x . s x j, j s 0, 1, 2, . . . , 2 n q 1, that is,

Ha x

b j

n

dxs

Ý ␻ i x ij ,

j s 0, 1, 2, . . . , 2 n q 1.

Ž 12.14 .

is0

This process produces 2 n q 2 equations to be solved for the ␻ i ’s and x i ’s. In particular, if the limits of integration are as y1, bs 1, then it can be shown Žsee Phillips and Taylor, 1973, page 140. that the x i-values will be the n q 1 zeros of the Legendre polynomial pnq1 Ž x . of degree n q 1 Žsee Section 10.2.. The ␻ i-values can be easily found by solving the system of equations Ž12.14., which is linear in the ␻ i ’s. For example, for n s 1, the zeros of p 2 Ž x . s 12 Ž3 x 2 y 1. are x 0 s y1r '3 , x 1 s 1r '3 . Applying Ž12.14., we obtain

Hy1 dxs ␻ 1

Hy1 x dxs ␻ 1

0

0 q ␻1

´

x 0 q ␻ 1 x1

´

'3 Ž y␻ 0 q ␻ 1 . s 0, 1 3

Hy1 x

2

dxs ␻ 0 x 02 q ␻ 1 x 12

´

Hy1 x

3

dxs ␻ 0 x 03 q ␻ 1 x 13

´

1

1

␻ 0 q ␻ 1 s 2, 1

Ž ␻ 0 q ␻ 1 . s 23 , 1

3'3

Ž y␻ 0 q ␻ 1 . s 0.

We note that the last two equations are identical to the first two. Solving the

525

GAUSSIAN QUADRATURE

latter for ␻ 0 and ␻ 1 , we get ␻ 0 s ␻ 1 s 1. Hence, we have the approximation

Hy1 f Ž x . dxf f 1

ž ' / ž' / y

1

1

qf

3

3

,

which is exact if f Ž x . is a polynomial of degree not exceeding 2 n q 1 s 3. If the limits of integration are not equal to y1, 1, we can easily convert the integral Hab f Ž x . dx to one with the limits y1, 1 by making the change of variable zs

2 xy Ž aq b . by a

.

This converts the general integral Hab f Ž x . dx to the integral wŽ b y 1 a.r2x Hy1 g Ž z . dz, where g Ž z. sf

Ž by a. zq bq a

.

2

We therefore have the approximation

Ha f Ž x . dxf b

by a 2

n

Ý ␻i g Ž zi . ,

Ž 12.15 .

is0

where the z i ’s are the zeros of the Legendre polynomial pnq1 Ž z .. It can be shown that Žsee Davis and Rabinowitz, 1975, page 75. that when as y1, bs 1, the error of approximation in Ž12.13. is given by

Ha

b

f Ž x . dxy

n

Ý ␻i f Ž x i . s

is0

Ž by a.

2 nq3

Ž n q 1. !

Ž 2 n q 3. Ž 2 n q 2. !

4

f Ž2 nq2. Ž ␰ . ,

3

a- ␰ - b,

Ž 12.16 . provided that f Ž2 nq2. Ž x . is continuous on w a, b x. This error decreases rapidly as n increases. Thus this Gaussian quadrature provides a very good approximation with a formula of the type given in Ž12.13.. There are several extensions of the approximation in Ž12.13.. These extensions are of the form n

Ha ␭Ž x . f Ž x . dxf Ý ␻ f Ž x . , b

i

is0

i

Ž 12.17 .

526

APPROXIMATION OF INTEGRALS

where ␭Ž x . is a particular positive weight function. As before, the coefficients ␻ 0 , ␻ 1 , . . . , ␻ n and the points x 0 , x 1 , . . . , x n , which belong to w a, b x, are chosen so that Ž12.17. is exact for all polynomials of degrees not exceeding 2 n q 1. The choice of the x i ’s depends on the form of ␭Ž x .. It can be shown that the values of x i are the zeros of a polynomial of degree n q 1 belonging to the sequence of polynomials that are orthogonal on w a, b x with respect to ␭Ž x . Žsee Davis and Rabinowitz, 1975, page 74; Phillips and Taylor, 1973, page 142.. For example, if as y1, bs 1, ␭Ž x . s Ž1 y x . ␣ Ž1 q x . ␤, ␣ ) y1, Ž ␣ , ␤ .Ž . Ž ␤ ) y1, then the x i ’s are the zeros of the Jacobi polynomial pnq1 x see 2 y1r2 Section 10.3.. Also, if as y1, bs 1, ␭Ž x . s Ž1 y x . , then the x i ’s are the zeros of the Chebyshev polynomial of the first kind, Tnq1 Ž x . Žsee Section 10.4., and so on. For those two cases, formula Ž12.17. is called the Gauss᎐Jacobi quadrature formula and the Gauss᎐Chebyshe® quadrature formula, respectively. The choice ␭Ž x . s 1 results in the original formula Ž12.13., which is now referred to as the Gauss᎐Legendre quadrature formula. EXAMPLE 12.4.1. Consider the integral H01 dxrŽ1 q x ., which has the exact value log 2 s 0.69314718. Applying formula Ž12.15., we get

H0

1

␻i

n

dx 1qx

f 12

Ý

is0

1 q Ž z i q 1. 1 2

␻i

n

s

Ý

is0

3 q zi

,

y1 F z i F 1,

Ž 12.18 .

where the z i ’s are the zeros of the Legendre polynomial pnq1 Ž z ., z s 2 xy 1. Let n s 1; then p 2 Ž z . s 12 Ž3 z 2 y 1. with zeros equal to z 0 s y1r '3 , z1 s 1r '3 . We have seen earlier that ␻ 0 s 1, ␻ 1 s 1; hence, from Ž12.18., we obtain

H0

1

dx 1qx

1

f

Ý

is0

s

␻i 3 q zi

'3 '3 q 3'3 y 1 3'3 q 1

s 0.692307691. Let us now use n s 2 in Ž12.18.. Then p 3 Ž z . s 12 Ž5 z 3 y 3 z . Žsee Section 10.2.. Its zeros are z 0 s yŽ 35 .1r2 , z1 s 0, z 2 s Ž 35 .1r2 . To find the ␻ i ’s, we apply formula Ž12.14. using as y1, bs 1, and z in place of x. For

527

GAUSSIAN QUADRATURE

j s 0, 1, 2, 3, 4, 5, we have

Hy1 dzs ␻ 1

Hy1 z dzs ␻

0 q ␻1 q ␻2 ,

1

0

z 0 q ␻ 1 z1 q ␻ 2 z 2 ,

Hy1 z

2

dzs ␻ 0 z 02 q ␻ 1 z12 q ␻ 2 z 22 ,

Hy1 z

3

dzs ␻ 0 z 03 q ␻ 1 z13 q ␻ 2 z 23 ,

Hy1 z

4

dzs ␻ 0 z 04 q ␻ 1 z14 q ␻ 2 z 24 ,

Hy1 z

5

dzs ␻ 0 z 05 q ␻ 1 z15 q ␻ 2 z 25 .

1

1

1

1

These equations can be written as

␻ 0 q ␻ 1 q ␻ 2 s 2, 1r2 Ž 35 . Ž y␻ 0 q ␻ 2 . s 0, 3 5

Ž ␻ 0 q ␻ 2 . s 23 ,

3r2 Ž 35 . Ž y␻ 0 q ␻ 2 . s 0, 9 25 9 25

Ž ␻ 0 q ␻ 2 . s 52 ,

1r2 Ž 53 . Ž y␻ 0 q ␻ 2 . s 0.

The above six equations can be reduced to only three that are linearly independent, namely,

␻ 0 q ␻ 1 q ␻ 2 s 2, y ␻ 0 q ␻ 2 s 0,

␻ 0 q ␻ 2 s 109 , the solution of which is ␻ 0 s 95 , ␻ 1 s 98 , ␻ 2 s 95 . Substituting the ␻ i ’s and z i ’s

528

APPROXIMATION OF INTEGRALS

in Ž12.18., we obtain

H0

dx

1

1qx

f

s

␻0 3 q z0

q

␻1 3 q z1

5 9

3 y Ž 35 .

1r2

q

8 9

q

q

3

␻2 3 q z2 5 9

3 q Ž 35 .

1r2

s 0.693121685. For higher values of n, the zeros of Legendre polynomials and the corresponding values of ␻ i can be found, for example, in Shoup Ž1984, Table 7.5. and Krylov Ž1962, Appendix A.. 12.5. APPROXIMATION OVER AN INFINITE INTERVAL Consider an integral of the form Ha⬁ f Ž x . dx, which is improper of the first kind Žsee Section 6.5.. It can be approximated by using the integral Hab f Ž x . dx for a sufficiently large value of b, provided, of course, that Ha⬁ f Ž x . dx is convergent. The methods discussed earlier in Sections 12.1᎐12.4 can then be applied to Hab f Ž x . dx. For improper integrals of the first kind, of the form H0⬁␭Ž x . f Ž x . dx, ⬁ Hy⬁ ␭Ž x . f Ž x . dx, we have the following Gaussian approximations: n

⬁

H0 ␭Ž x . f Ž x . dxf Ý ␻ f Ž x . , i

i

Ž 12.19 .

is0

⬁

Hy⬁

␭Ž x . f Ž x . dxf

n

Ý ␻i f Ž x i . ,

Ž 12.20 .

is0

where, as before, the x i ’s and ␻ i ’s are chosen so that Ž12.19. andŽ12.20. are exact for all polynomials of degrees not exceeding 2 n q 1. For the weight function ␭Ž x . in Ž12.19., the choice ␭Ž x . s eyx gives the Gauss᎐Laguerre quadrature, for which the x i ’s are the zeros of the Laguerre polynomial L nq1 Ž x . of degree n q 1 and ␣ s 0 Žsee Section 10.6.. The associated error of approximation is given by Žsee Davis and Rabinowitz, 1975, page 173. ⬁

H0

Ž n q 1 . ! Ž2 nq2. e f Ž x . dxy Ý ␻ i f Ž x i . s f Ž ␰ ., Ž 2 n q 2. ! is0 n

yx

2

0 - ␰ - ⬁.

Choosing ␭Ž x . s eyx in Ž12.20. gives the Gauss᎐Hermite quadrature, and the corresponding x i ’s are the zeros of the Hermite polynomial Hnq1 Ž x . of degree n q 1 Žsee Section 10.5.. The associated error of approximation is of the form Žsee Davis and Rabinowitz, 1975, page 174. 2

⬁

Hy⬁e

yx 2

f Ž x . dxy

n Ž n q 1 . !'␲ Ý ␻ i f Ž x i . s 2 nq1 Ž 2 n q 2. ! f Ž2 nq2. Ž ␰ . ,

is0

y⬁ - ␰ - ⬁.

529

APPROXIMATION OVER AN INFINITE INTERVAL

We can also use the Gauss-Laguerre and the Gauss᎐Hermite quadrature formulas to approximate convergent integrals of the form H0⬁ f Ž x . dx, ⬁ Hy⬁ f Ž x . dx: ⬁

H0

f Ž x . dxs

⬁

H0 e

yx

e x f Ž x . dx

n

f

Ý ␻i e x f Ž x i . , i

is0 ⬁

⬁

Hy⬁ f Ž x . dxs Hy⬁ e

e f Ž x . dx

yx 2 x 2

n

f

Ý ␻i e x f Ž x i . . 2 i

is0

EXAMPLE 12.5.1.

Consider the integral Is

⬁

H0

eyx x 1 y ey2 x

dx

n

f

Ý ␻i f Ž x i . ,

Ž 12.21 .

is0

where f Ž x . s x Ž1 y ey2 x .y1 and the x i ’s are the zeros of the Laguerre polynomial L nq1 Ž x .. To find expressions for L nŽ x ., n s 0, 1, 2, . . . , we can use the recurrence relation Ž10.37. with ␣ s 0, which can be written as L nq1 Ž x . s Ž xy n y 1 . L n Ž x . y x

dLn Ž x . dx

n s 0, 1, 2, . . . . Ž 12.22 .

,

Recall that L0 Ž x . s 1. Choosing n s 1 in Ž12.21., we get I f ␻ 0 f Ž x 0 . q ␻ 1 f Ž x1 . .

Ž 12.23 .

From Ž12.22. we have L1 Ž x . s Ž xy 1 . L0 Ž x . s xy 1, L2 Ž x . s Ž xy 2 . L1 Ž x . y x

d Ž xy 1 . dx

s Ž xy 2 . Ž xy 1 . y x s x 2 y 4 xq 2. The zeros of L2 Ž x . are x 0 s 2 y '2 , x 1 s 2 q '2 . To find ␻ 0 and ␻ 1 , formula Ž12.19. must be exact for all polynomials of degrees not exceeding

530

APPROXIMATION OF INTEGRALS

2 n q 1 s 3. This is equivalent to requiring that ⬁

H0 e ⬁

yx

H0 e ⬁

yx

H0 e ⬁

yx

H0 e

dxs ␻ 0 q ␻ 1 ,

x dxs ␻ 0 x 0 q ␻ 1 x 1 ,

x 2 dxs ␻ 0 x 02 q ␻ 1 x 12 ,

yx

x 3dxs ␻ 0 x 03 q ␻ 1 x 13 .

Only two equations are linearly independent, the solution of which is ␻ 0 s 0.853553, ␻ 1 s 0.146447. From Ž12.23. we then have ␻ 0 x0 ␻ 1 x1 If q y2 x 0 1ye 1 y ey2 x 1 s 1.225054. Let us now calculate Ž12.21. using n s 2, 3, 4. The zeros of Laguerre polynomials L3 Ž x ., L4Ž x ., L5 Ž x ., and the values of ␻ i for each n are shown in Table 12.1. These values are given in Ralston and Rabinowitz Ž1978, page 106. and also in Krylov Ž1962, Appendix C.. The corresponding approximate values of I are given in Table 12.1. It can be shown that the exact value of I is ␲ 2r8 f 1.2337.

Table 12.1. Zeros of Laguerre Polynomials ( x i ), Values of ␻ i , and the Corresponding Approximate Values a of I n

xi

␻i

I

1

0.585786 3.414214

0.853553 0.146447

1.225054

2

0.415775 2.294280 6.289945

0.711093 0.278518 0.010389

1.234538

0.322548 1.745761 4.536620 9.395071

0.603154 0.357419 0.038888 0.000539

1.234309

0.263560 1.413403 3.596426 7.085810 12.640801

0.521756 0.398667 0.075942 0.003612 0.000023

1.233793

3

4

a

See Ž12.21..

531

THE METHOD OF LAPLACE

12.6. THE METHOD OF LAPLACE This method is used to approximate integrals of the form I Ž ␭. s

Ha ␸ Ž x . e b

␭ hŽ x .

Ž 12.24 .

dx,

where ␭ is a large positive constant, ␸ Ž x . is continuous on w a, b x, and the first and second derivatives of hŽ x . are continuous on w a, b x. The limits a and b may be finite or infinite. This integral was used by Laplace in his original development of the central limit theorem Žsee Section 4.5.1.. More specifically, if X 1 , X 2 , . . . , X n , . . . is a sequence of independent and identically distributed random variables with a common density function, then the density function of the sum Sn s Ý nis1 X i can be represented in the form Ž12.24. Žsee Wong, 1989, Chapter 2.. Suppose that hŽ x . has a single maximum in the interval w a, b x at xs t, aF t - b, where hX Ž t . s 0 and hY Ž t . - 0. Hence, e ␭ hŽ x . is maximized at t for any ␭ ) 0. Suppose further that e ␭ hŽ x . becomes very strongly peaked at xs t and decreases rapidly away from xs t on w a, b x as ␭ ™ ⬁. In this case, the major portion of I Ž ␭. comes from integrating the function ␸ Ž x . e ␭ hŽ x . over a small neighborhood around xs t. Under these conditions, it can be shown that if a- t - b, and as ␭ ™ ⬁, I Ž ␭ . ; ␸ Ž t . e ␭ hŽ t .

y2␲

1r2

,

␭ hY Ž t .

Ž 12.25 .

where ; denotes asymptotic equality Žsee Section 3.3.. Formula Ž12.25. is known as Laplace’s approximation. A heuristic derivation of Ž12.25. can be arrived at by replacing ␸ Ž x . and hŽ x . by the leading terms in their Taylor’s series expansions around xs t. The integration limits are then extended to y⬁ and ⬁, that is,

Ha ␸ Ž x . e b

␭ hŽ x .

dxf f

Ha ␸ Ž t . exp b

␭hŽ t . q

⬁

Hy⬁␸ Ž t . exp

s ␸ Ž t . e ␭ hŽ t . s ␸ Ž t . e ␭ hŽ t .

␭ 2

␭hŽ t . q

⬁

Hy⬁exp y2␲

␭ hY Ž t .

␭ 2

2 Ž xy t . hY Ž t . dx

␭ 2

2 Ž x y t . hY Ž t . dx

2 Ž xy t . hY Ž t . dx

Ž 12.26 .

1r2

.

Ž 12.27 .

Formula Ž12.27. follows from Ž12.26. by making use of the fact that H0⬁eyx dx s 12 ⌫ Ž 12 . s '␲ r2, where ⌫ Ž⭈. is the gamma function Žsee Example 6.9.6., or by simply evaluating the integral of a normal density function. 2

532

APPROXIMATION OF INTEGRALS

If t s a, then it can be shown that as ␭ ™ ⬁, I Ž ␭ . ; ␸ Ž a . e ␭ hŽ a.

y␲

1r2

.

Y

2 ␭ h Ž a.

Ž 12.28 .

Rigorous proofs of Ž12.27. and Ž12.28. can be found in Wong Ž1989, Chapter 2., Copson Ž1965, Chapter 5., Fulks Ž1978, Chapter 18., and Lange Ž1999, Chapter 4.. EXAMPLE 12.6.1.

Consider the gamma function ⌫ Ž n q 1. s

⬁

H0 e

yx

n ) y1.

x n dx,

Ž 12.29 .

Let us find an approximation for ⌫ Ž n q 1. when n is large an positive, but not necessarily an integer. Let xs nz; then Ž12.29. can be written as ⬁

H0 e

⌫ Ž n q 1. s n

⬁

yn z

H0 e

sn

s n nq1

yn z

exp nlog Ž nz . dz exp w n log n q n log z x dz

⬁

H0 exp

n Ž yzq log z . dz.

Ž 12.30 .

Let hŽ z . s yzq log z. Then hŽ z . has a unique maximum at z s 1 with hX Ž1. s 0 and hY Ž1. s y1. Applying formula Ž12.27. to Ž12.30., we obtain ⌫ Ž n q 1. ; n

nq1 yn

e

y2␲

1r2

n Ž y1 .

s eyn n n'2␲ n ,

Ž 12.31 .

as n ™ ⬁. Formula Ž12.31. is known as Stirling’s formula. EXAMPLE 12.6.2.

Consider the integral, In Ž ␭ . s

1

␲

H exp Ž ␭ cos x . cos nx dx, ␲ 0

as ␭ ™ ⬁. This integral looks like Ž12.24. with hŽ x . s cos x, which has a single maximum at xs 0 in w0, ␲ x. Since hY Ž0. s y1, then by applying Ž12.28.

533

MULTIPLE INTEGRALS

we obtain, as ␭ ™ ⬁, In Ž ␭ . ; s EXAMPLE 12.6.3.

1

␲

y␲

e␭

1r2

2 ␭Ž y1 .

e␭

.

'2␲␭

Consider the integral I Ž ␭. s

␲

H0 exp w ␭ sin x x dx

as ␭ ™ ⬁. Here, hŽ x . s sin x, which has a single maximum at x s ␲r2 in w0, ␲ x, and hY Ž␲r2. s y1. From Ž12.27., we get y2␲

I Ž ␭. ; e ␭

1r2

␭Ž y1 .

(

se␭

2␲

␭

as ␭ ™ ⬁.

12.7. MULTIPLE INTEGRALS We recall that integration of a multivariable function was discussed in Section 7.9. In the present section, we consider approximate integration formulas for an n-tuple Riemann integral over a region D in an n-dimensional Euclidean space R n. For example, let us consider the double integral I s HHD f Ž x 1 , x 2 . dx 1 dx 2 , where D ; R 2 is the region D s
Ž x 1 , x 2 . aF x 1 F b, ␺ Ž x 1 . F x 2 F ␾ Ž x 1 . 4 . Then Is

Ha H␺␾Ž xx. f Ž x , x b

Ž

1.

1

2

. dx 2 dx1

1

s

Ha g Ž x . dx , b

1

Ž 12.32 .

1

where g Ž x1 . s

H␺␾Ž xx. f Ž x , x Ž

1.

1

1

2

. dx 2 .

Ž 12.33 .

534

APPROXIMATION OF INTEGRALS

Let us now apply a Gaussian integration rule to Ž12.32. using the points x 1 s z 0 , z1 , . . . , z m with the matching coefficients ␻ 0 , ␻ 1 , . . . , ␻ m , m

Ha g Ž x . dx f Ý ␻ g Ž z . . b

1

i

1

i

is0

Thus m

If

Ý ␻ iH Ž

␾ Ž zi.

␺ z i.

is0

f Ž z i , x 2 . dx 2 .

Ž 12.34 .

For the ith of the m q 1 integrals in Ž12.34. we can apply a Gaussian integration rule using the points yi0 , yi1 , . . . , yi n with the corresponding coefficients ®i0 , ®i1 , . . . , ®i n . We then have

H␺␾Ž zz. f Ž z , x Ž i.

i

2

. dx 2 f

n

Ý ®i j f Ž z i , yi j . .

js0

i

Hence, m

Is

n

Ý Ý ␻ i ®i j f Ž z i , yi j . .

is0 js0

This procedure can obviously be generalized to higher-order multiple integrals. More details can be found in Stroud Ž1971. and Davis and Rabinowitz Ž1975, Chapter 5.. The method of Laplace in Section 12.6 can be extended to an n-dimensional integral of the form I Ž ␭. s

HD␸ Ž x. e

␭ hŽx.

dx,

which is a multidimensional version of the integral in Ž12.24.. Here, D is a region in R n, which may be bounded or unbounded, ␭ is a large positive constant, and x s Ž x 1 , x 2 , . . . , x n .X . As before, it is assumed that: a. ␸ Ž x. is continuous in D. b. hŽx. has continuous first-order and second-order partial derivatives with respect to x 1 , x 2 , . . . , x n in D. c. hŽx. has a single maximum in D at x s t. If t is an interior point of D, then it is also a stationary point of hŽx., that is, ⭸ hr⭸ x i < xst s 0, i s 1, 2, . . . , n, since t is a point of maximum and the

535

THE MONTE CARLO METHOD

partial derivatives of hŽx. exist. Furthermore, the Hessian matrix, H h Ž t. s

ž

⭸ 2 h Ž x. ⭸ xi⭸ x j

/

, xst

is negative definite. Then, for large ␭, I Ž ␭. is approximately equal to I Ž ␭. ;

ž / 2␲

nr2

␭

␸ Ž t .
ydet H h Ž t .

4 y1r2 e ␭ hŽt. .

A proof of this approximation can be found in Wong Ž1989, Section 9.5.. We note that this expression is a generalization of Laplace’s formula Ž12.25. to an n-tuple Riemann integral. If t happens to be on the boundary of D and still satisfies the conditions that ⭸ hr⭸ x i < xst s 0 for i s 1, 2, . . . , n, and H hŽt. is negative definite, then it can be shown that for large ␭, I Ž ␭. ;

ž /

1 2␲ 2

␭

nr2

␸ Ž t .
ydet H h Ž t .

4 y1r2 e ␭ hŽt. ,

which is one-half of the previous approximation for I Ž ␭. Žsee Wong,1989, page 498..

12.8. THE MONTE CARLO METHOD A new approach to approximate integration arose in the 1940s as part of the Monte Carlo method of S. Ulam and J. von Neumann ŽHaber, 1970.. The basic idea of the Monte Carlo method for integrals is described as follows: suppose that we need to compute the integral Is

Ha f Ž x . dx. b

Ž 12.35 .

We consider I as the expected value of a certain stochastic process. An estimate of I can be obtained by random sampling from this process, and the estimate is then used as an approximation to I. For example, let X be a continuous random variable that has the uniform distribution U Ž a, b . over the interval w a, b x. The expected value of f Ž X . is E fŽ X. s s

1 by a I by a

Ha f Ž x . dx b

.

536

APPROXIMATION OF INTEGRALS

Let x 1 , x 2 , . . . , x n be a random sample from U Ž a, b .. An estimate of E w f Ž X .x is given by Ž1rn.Ý nis1 f Ž x i .. Hence, an approximate value of I, denoted by Iˆn , can be obtained as Iˆn s

n

by a

Ý f Ž xi . .

n

Ž 12.36 .

is1

The justification for using Iˆn as an approximation to I is that Iˆn is a consistent estimator of I, that is, for a given ⑀ ) 0, lim P Ž Iˆn y I G ⑀ . s 0.

n™⬁

This is true because Ž1rn.Ý nis1 f Ž x i . converges in probability to E w f Ž X .x as n ™ ⬁, according to the law of large numbers Žsee Sections 3.7 and 5.6.3.. In other words, the probability that Iˆn will be different from I can be made arbitrarily close to zero if n is chosen large enough. In fact, we even have the stronger result that Iˆn converges strongly, or almost surely, to I, as n ™ ⬁, by the strong law of large numbers Žsee Section 5.6.3.. The closeness of Iˆn to I depends on the variance of Iˆn , which is equal to Var Ž Iˆn . s Ž by a . Var 2

s

2 Ž by a. ␴f 2

n

n

1 n

Ý f Ž xi .

is1

Ž 12.37 .

,

where ␴f 2 is the variance of the random variable f Ž X ., that is,

␴f 2 s Var f Ž X . sE f 2 Ž X . y
E f Ž X . s

1

Hf by a a

b 2

Ž x . dxy

ž

42 I

by a

/

2

Ž 12.38 .

.

By the central limit theorem Žsee Section 4.5.1., if n is large enough, then Iˆn is approximately normally distributed with mean I and variance Ž1rn.Ž b y a. 2␴f 2 . Thus, Iˆn y I

Z,

6

Ž by a. ␴fr'n

d

d

denotes

6

where Z has the standard normal distribution, and the symbol

537

THE MONTE CARLO METHOD

convergence in distribution Žsee Section 4.5.1.. It follows that for a given ␶ ) 0, P

Iˆn y I F

␶

Ž by a. ␴f f

'n

1

␶

'2␲ Hy␶e

yx 2 r2

dx.

Ž 12.39 .

The right-hand side of Ž12.39. is the probability that a standard normal distribution attains values between y␶ and ␶ . Let us denote this probability by 1 y ␣ . Then ␶ s z␣ r2 , which is the upper Ž ␣r2.100th percentile of the standard normal distribution. If we denote the error of approximation, Iˆn y I, by En , then formula Ž12.39. indicates that for large n, En F

1

'n Ž by a. ␴f z␣ r2

Ž 12.40 .

with an approximate probability equal to 1 y ␣ , which is called the confidence coefficient. Thus, for a fixed ␣ , the error bound in Ž12.40. is proportional to ␴f and is inversely proportional to 'n . For example, if 1 y ␣ s 0.90, then z␣ r2 s 1.645, and En F

1.645

'n Ž by a. ␴f

with an approximate confidence coefficient equal to 0.90. Also, if 1 y ␣ s 0.95, then z␣ r2 s 1.96, and En F

1.96

'n Ž by a. ␴f

with an approximate confidence coefficient equal to 0.95. In order to compute the error bound in Ž12.40., an estimate of ␴f 2 is needed. Using Ž12.38. and the random sample x 1 , x 2 , . . . , x n , an estimate of ␴f 2 is given by

␴f 2n s

ˆ

1 n

n

Ý f Ž xi . y 2

is1

1 n

n

Ý f Ž xi .

2

.

Ž 12.41 .

is1

12.8.1. Variance Reduction In order to increase the accuracy of the Monte Carlo approximation of I, the error bound in Ž12.40. should be reduced for a fixed value of ␣ . We can achieve this by increasing n. Alternatively, we can reduce ␴f by considering a distribution other than the uniform distribution. This can be accomplished by using the so-called method of importance sampling, a description of which follows.

538

APPROXIMATION OF INTEGRALS

Let g Ž x . be a density function that is positive over the interval w a, b x. Thus, g Ž x . ) 0, aF xF b, and Hab g Ž x . dxs 1. The integral in Ž12.35. can be written as Is

Ha

b

f Ž x. gŽ x.

g Ž x . dx.

Ž 12.42 .

In this case, I is the expected value of f Ž X .rg Ž X ., where X is a continuous random variable with the density function g Ž x .. Using now a random sample, x 1 , x 2 , . . . , x n , from this distribution, an estimate of I can be obtained as IˆnU s

1

n

f Ž xi .

. Ý n is1 g Ž x i .

Ž 12.43 .

The variance of IˆnU is then given by Var Ž IˆnU . s

␴f 2g n

,

where

␴f 2g s Var

½

sE

s

Ha

b

fŽ X. gŽ X . fŽ X. gŽ X .

f 2Ž x. g2Ž x.

2

5 ½

y E

fŽ X. gŽ X .

g Ž x . dxy I 2 .

5

2

Ž 12.44 .

As before, the error bound can be derived on the basis of the central limit theorem, using the fact that IˆnU is approximately normally distributed with mean I and variance Ž1rn. ␴f 2g for large n. Hence, as in Ž12.40., EnU F

1

'n

␴f g z␣ r2

with an approximate probability equal to 1 y ␣ , where EnU s IˆnU y I. The density g Ž x . should therefore be chosen so that an error bound smaller than

539

THE MONTE CARLO METHOD

Table 12.2. Approximate Values of I s H12 x 2 dx Using Formula (12.36) n

Iˆn

50 100 150 200 250 300 350

2.3669 2.5087 2.2227 2.3067 2.3718 2.3115 2.3366

the one in Ž12.40. can be achieved. For example, if f Ž x . ) 0, and if gŽ x. s

f Ž x.

Ha

b

,

f Ž x . dx

Ž 12.45 .

then ␴f 2g s 0 as can be seen from formula Ž12.44.. Unfortunately, since the exact value of Hab f Ž x . dx is the one we seek to compute, formula Ž12.45. cannot be used. However, by choosing g Ž x . to behave approximately as f Ž x . wassuming f Ž x . is positivex, we should expect a reduction in the variance. Note that the generation of random variables from the g Ž x . distribution is more involved than just using a random sample from the uniform distribution U Ž a, b .. EXAMPLE 12.8.1. Consider the integral I s H12 x 2 dxs 73 f 2.3333. Suppose that we use a sample of 50 points from the uniform distribution U Ž1, 2.. Applying formula Ž12.36., we obtain Iˆ50 s 2.3669. Repeating this process several times using higher values of n, we obtain the values in Table 12.2. EXAMPLE 12.8.2. Let us now apply the method of importance sampling to the integral I s H01 e x dxf 1.7183. Consider the density function g Ž x . s 2 Ž . w x 3 1 q x over the interval 0, 1 . Using the method described in Section 3.7, a random sample, x 1 , x 2 , . . . , x n , can be generated from this distribution as follows: the cumulative distribution function for g Ž x . is y s G Ž x . s P w X F x x, that is, ysGŽ x. s s 23 s

H0 g Ž t . dt x

H0 Ž 1 q t . dt

2 3

x

ž

xq

x2 2

/

,

0 F x F 1.

540

APPROXIMATION OF INTEGRALS

Table 12.3. Approximate Values of I s H01e x dx n

IˆnU wFormula Ž12.46.x

Iˆn wFormula Ž12.36.x

50 100 150 200 250 300 350 400

1.7176 1.7137 1.7063 1.7297 1.7026 1.7189 1.7093 1.7188

1.7156 1.7025 1.6854 1.7516 1.6713 1.7201 1.6908 1.7192

The only solution of y s GŽ x . in w0, 1x is xs y1 q Ž 1 q 3 y .

1r2

0 F y F 1.

,

Hence, the inverse function of G Ž x . is Gy1 Ž y . s y1 q Ž 1 q 3 y .

1r2

0 F y F 1.

,

If y 1 , y 2 , . . . , yn form a random sample of n values from the uniform distribution U Ž0, 1., then x 1 s Gy1 Ž y 1 ., x 2 s Gy1 Ž y 2 ., . . . , x n s Gy1 Ž yn . will form a sample from the distribution with the density function g Ž x .. Formula Ž12.43. can then be applied to approximate the value of I using the estimate IˆnU s

3 2n

n

Ý

is1

e xi 1 q xi

.

Ž 12.46 .

Table 12.3 gives IˆnU for several values of n. For the sake of comparison, values of Iˆn from formula Ž12.36. wwith f Ž x . s e x , as 0, bs 1x were also computed using a sample from the uniform distribution U Ž0, 1.. The results are shown in Table 12.3. We note that the values of IˆnU are more stable and closer to the true value of I than those of Iˆn . 12.8.2. Integrals in Higher Dimensions The Monte Carlo method can be extended to multidimensional integrals. Consider computing the integral I s HD f Žx. dx, where D is a bounded region in R n, the n-dimensional Euclidean space, and x s Ž x 1 , x 2 , . . . , x n .X . As before, we consider I as the expected value of a stochastic process having a certain distribution over D. For example, we may take X to be a continuous random vector uniformly distributed over D. By this we mean that the probability of X being in D is 1r®Ž D ., where ®Ž D . denotes the volume of D,

541

APPLICATIONS IN STATISTICS

and the probability of X being outside D is zero. Hence, the expected value of f ŽX. is 1

E f Ž X. s

®Ž D . I

s

®Ž D .

HD f Ž x. dx .

The variance of f ŽX., denoted by ␴f 2 , is

␴f 2 s E f 2 Ž X . y
E f Ž X . s

1 ®Ž D .

HD f

2

Ž x . dx y

42 I ®Ž D .

2

.

Let us now take a sample of N independent observations on X, namely, N X 1 , X 2 , . . . , X N . Then a consistent estimator of E w f ŽX.x is Ž1rN .Ý is1 f ŽX i ., and hence, an estimate of I is given by IˆN s

®Ž D . N

N

Ý f ŽX i . ,

is1

which can be used to approximate I. The variance of IˆN is Var Ž IˆN . s

␴f 2 N

®2 Ž D . .

Ž 12.47 .

If N is large enough, then by the central limit theorem, IˆN is approximately normally distributed with mean I and variance as in Ž12.47.. It follows that EN F

®Ž D .

'N

␴f z␣ r2

with an approximate probability equal to 1 y ␣ , where EN s I y IˆN is the error of approximation. This formula is analogous to formula Ž12.40.. The method of importance sampling can also be applied here to reduce the error of approximation. The application of this method is similar to the case of a single-variable integral as seen earlier.

12.9. APPLICATIONS IN STATISTICS Approximation of integrals is a problem of substantial concern for statisticians. The statistical literature in this area has grown significantly in the last 20 years, particularly in connection with integrals that arise in Bayesian

542

APPROXIMATION OF INTEGRALS

statistics. Evans and Swartz Ž1995. presented a survey of the major techniques and approaches available for the numerical approximation of integrals in statistics. The proceedings edited by Flournoy and Tsutakawa Ž1991. includes several interesting articles on statistical multiple integration, including a detailed description of available software to compute multidimensional integrals Žsee the article by Kahaner, 1991, page 9.. 12.9.1. The Gauss-Hermite Quadrature The Gauss᎐Hermite quadrature mentioned earlier in Section 12.5 is often used for numerical integration in statistics because of its relation to Gaussian Žnormal. densities. We recall that this quadrature is defined in terms of 2 ⬁ integrals of the form Hy⬁ eyx f Ž x . dx. Using formula Ž12.20., we have approximately ⬁

Hy⬁e

yx 2

f Ž x . dxf

n

Ý ␻i f Ž x i . ,

is0

where the x i ’s are the zeros of the Hermite polynomial Hnq1 Ž x . of degree n q 1, and the ␻ i ’s are suitably corresponding weights. Tables of x i and ␻ i values are given by Abramowitz and Stegun Ž1972, page 924. and by Krylov Ž1962, Appendix B.. Liu and Pierce Ž1994. applied the Gauss᎐Hermite quadrature to integrals ⬁ of the form Hy⬁ g Ž t . dt, which can be expressed in the form ⬁

⬁

Hy⬁ g Ž t . dts Hy⬁ f Ž t . ␾ Ž t , ␮ , ␴ . dt, where ␾ Ž t, ␮ , ␴ . is the normal density

␾ Ž t, ␮, ␴ . s

1

'2␲␴ 2

exp y

1 2␴ 2

2 Ž ty␮. ,

and f Ž t . s g Ž t .r␾ Ž t, ␮ , ␴ .. Thus, ⬁

1

⬁

Hy⬁ g Ž t . dts Hy⬁ '2␲␴ s

f

1

'␲ 1

⬁

2

f Ž t . exp y

1 2␴ 2

Hy⬁ f Ž ␮ q '2 ␴ x . e n

yx 2

Ý ␻ i f Ž ␮ q '2 ␴ x i . , '␲ is0

2 Ž t y ␮ . dt

dx

Ž 12.48 .

543

APPLICATIONS IN STATISTICS

where the x i ’s are the zeros of the Hermite polynomial Hnq1 Ž x . of degree n q 1. We may recall that the x i ’s and ␻ i ’s are chosen so that this approximation will be exact for all polynomials of degrees not exceeding 2 n q 1. For this reason, Liu and Pierce Ž1994. recommend choosing ␮ and ␴ in Ž12.48. so that f Ž t . is well approximated by a low-order polynomial in the region where the values of ␮ q '2 ␴ x i are taken. More specifically, the Ž n q 1.thorder Gauss᎐Hermite quadrature in Ž12.48. will be highly effective if the ratio of g Ž t . to the normal density ␾ Ž t, ␮ , ␴ 2 . can be well approximated by a polynomial of degree not exceeding 2 n q 1 in the region where g Ž t . is substantial. This arises frequently, for example, when g Ž t . is a likelihood function wif g Ž t . ) 0x, or the product of a likelihood function and a normal density, as was pointed out by Liu and Pierce Ž1994., who gave several examples to demonstrate the usefulness of the approximation in Ž12.48.. 12.9.2. Minimum Mean Squared Error Quadrature Correlated observations may arise in some experimental work ŽPiegorsch and Bailer, 1993.. Consider, for example, the model yq r s f Ž t q . q ⑀ q r , where yq r represents the observed value from experimental unit r at time t q Ž q s 0, 1, . . . , m; r s 1, 2, . . . , n., f Ž t q . is the underlying response function, and ⑀ q r is a random error term. It is assumed that E Ž ⑀ q r . s 0, CovŽ ⑀ p r , ⑀ q r . s ␴p q , and CovŽ ⑀ p r , ⑀ q s . s 0 Ž r / s . for all p, q. The area under the response curve over the interval t 0 - t - t m is As

Ht

tm

f Ž t . dt.

Ž 12.49 .

0

This is an important measure for a variety of experimental situations, including the assessment of chemical bioavailability in drug disposition studies ŽGibaldi and Perrier, 1982, Chapters 2, 7. and other clinical settings. If the functional form of f Ž⭈. is unknown, the integral in Ž12.49. is estimated by numerical methods. This is accomplished using a quadrature approximation of the integral in Ž12.49.. By definition, a quadrature estimator of this integral is m

Aˆs

Ý ␾q fˆq ,

Ž 12.50 .

qs0

where fˆq is some unbiased estimator of f q s f Ž t q . with CovŽ fˆp , fˆq . s ␴p qrn, and the ␾ q ’s form a set of quadrature coefficients.

544

APPROXIMATION OF INTEGRALS

ˆ The expected value of A, E Ž Aˆ. s

m

Ý ␾q f q ,

qs0

may not necessarily be equal to A due to the quadrature approximation ˆ The bias in estimating A is employed in calculating A. bias s E Ž Aˆ. y A, and the mean squared error ŽMSE. of Aˆ is given by MSE Ž Aˆ. s Var Aˆq E Ž Aˆ. y A m

s

m

Ý Ý ␾ p ␾q

ps0 qs0

␴p q n

2

žÝ

/

m

q

2

␾q f q y A ,

qs0

which can be written as MSE Ž Aˆ. s

1 n

␾X V␾ q Ž f X ␾ y A . , 2

where Vs Ž ␴p q ., f s Ž f 0 , f 1 , . . . , f m .X , and ␾X s Ž ␾ 0 , ␾ 1 , . . . , ␾m .. Hence, MSE Ž Aˆ. s ␾X

1 n

Vq f f X ␾ y 2 Af X ␾ q A2 .

Ž 12.51 .

Let us now seek an optimum value of ␾ that minimizes MSEŽ Aˆ. in Ž12.51.. For this purpose, we equate the gradient of MSEŽ Aˆ. with respect to ␾, namely ⵱␾ MSEŽ Aˆ., to zero. We obtain ⵱␾ MSE Ž Aˆ. s 2

ž

1 n

/

Vq f f X ␾ y Af s 0.

Ž 12.52 .

In order for this equation to have a solution, the vector Af must belong to the column space of the matrix Ž1rn.Vq f f X . Note that this matrix is positive semidefinite. If its inverse exists, then it will be positive definite, and the only solution, ␾U , to Ž12.52., namely, ␾U s A

ž

1 n

Vq f f X

/

y1

f,

Ž 12.53 .

yields a unique minimum of MSEŽ Aˆ. Žsee Section 7.7.. Using ␾U in Ž12.50.,

545

APPLICATIONS IN STATISTICS

we obtain the following estimate of A: AˆU s ˆ f X ␾U 1

s AfˆX

n

y1

Vq f f

X

Ž 12.54 .

f.

Using the Sherman᎐Morrison formula Žsee Exercise 2.14., Ž12.54. can be written as AˆU s nA ˆ f X Vy1 f y

nfˆX Vy1 f f X Vy1 f 1 q nf X Vy1 f

,

Ž 12.55 .

where ˆ f s Ž fˆ0 , fˆ1 , . . . , fˆm .X . We refer to AˆU as an MSE-optimal estimator of A. Replacing f with its unbiased estimator ˆ f, and A with some initial estimate, say A 0 s ˆ f X ␾ 0 , where ␾ 0 is an initial value of ␾, yields the approximate MSE-optimal estimator AˆUU s

c 1qc

Ž 12.56 .

A0 ,

where c is the quadratic form, c s nfˆX Vy1 ˆ f.

˜ where ␾ ˜ s w crŽ1 q c .x␾ 0 . Since c G 0, AˆUU This estimator has the form ˆ f X ␾, provides a shrinkage of the initial estimate in A 0 toward zero. This creates a biased estimator of A with a smaller variance, which results in an overall reduction in MSE. A similar approach is used in ridge regression to estimate the parameters of a linear model when the columns of the corresponding matrix X are multicollinear Žsee Section 5.6.6.. We note that this procedure requires knowledge of V. In many applications, it may not be possible to specify V. Piegorsch and Bailer Ž1993. used an estimate of V when V was assumed to have certain particular patterns, such as Vs ␴ 2 I or Vs ␴ 2 wŽ1 y ␳ .I q ␳ Jx, where I and J are, respectively, the identity matrix and the square matrix of ones, and ␴ 2 and ␳ are unknown constants. For example, under the equal variance assumption Vs ␴ 2 I, the ratio crŽ1 q c . is equal to c 1qc

ž

s 1q

␴2 nfˆX ˆ f

/

y1

.

If ˆ f is normally distributed, ˆ f ; N wf, Ž1rn.Vx ᎏas is the case when ˆ f is a vector ˆ Ž of means, f s y 0 , y 1 , . . . , ym .X with yq s Ž1rn.Ý nrs1 yq r ᎏthen an unbiased

546

APPROXIMATION OF INTEGRALS

estimate of ␴ 2 is given by s2 s

m

1

n

Ý Ý Ž y yy . Ž m q 1 . Ž n y 1 . qs0 rs1 q r q

2

.

Substituting s 2 in place of ␴ 2 in Ž12.56., we obtain the area estimator

ž

AˆUU s A 0 1 q

s2 nfˆX ˆ f

/

y1

.

A similar quadrature approximation of A in Ž12.49. was considered earlier by Katz and D’Argenio Ž1983. using a trapezoidal approximation to A. The quadrature points were selected so that they minimize the expectation of the square of the difference between the exact integral and the quadrature approximation. This approach was applied to simulated pharmacokinetic problems. 12.9.3. Moments of a Ratio of Quadratic Forms Consider the ratio of quadratic forms, Qs

y XAy y X By

,

Ž 12.57 .

where A and B are symmetric matrices, B is positive definite, and y is an n = 1 random vector. Ratios such as Q are frequently encountered in statistics and econometrics. In general, their exact distributions are mathematically intractable, especially when the quadratic forms are not independent. For this reason, the derivation of the moments of such ratios, for the purpose of approximating their distributions, is of interest. Sutradhar and Bartlett Ž1989. obtained approximate expressions for the first four moments of the ratio Q for a normally distributed y. The moments were utilized to approximate the distribution of Q. This approximation was then applied to calculate the percentile points of a modified F-test statistic for testing treatment effects in a one-way model under correlated observations. MorinŽ1992. derived exact, but complicated, expressions for the first four moments of Q for a normally distributed y. The moments are expressed in terms of confluent hypergeometric functions of many variables. If y is not normally distributed, then no tractable formulas exist for the moments of Q in Ž12.57.. Hence, manageable and computable approximations for these moments would be helpful. Lieberman Ž1994. used the method of Laplace to provide general approximations for the moments of Q without making the normality assumption on y. Lieberman showed that if E Žy X By. and E wŽy XAy. k x exist for k G 1, then the Laplace approximation of

547

APPLICATIONS IN STATISTICS

E Ž Q k ., the kth noncentral moment of Q, is given by EL Ž Q

k

.s

E Ž y XAy .

k

E Ž y X By .

k

Ž 12.58 .

.

In particular, if y ; N Ž ␮ , ⌺ ., then the Laplace approximation for the mean and second noncentral moment of Q are written explicitly as EL Ž Q . s EL Ž Q

2

.s

tr Ž A ⌺ . q ␮X A␮ tr Ž B⌺ . q ␮X B␮ E Ž y XAy.

2

E Ž y X By .

2

s

Var Ž y XAy . q E Ž y XAy.

s

2tr Ž A ⌺ .

E Ž y X By . 2

2

2

q 4␮X A ⌺ A␮ q tr Ž A ⌺ . q ␮X A␮ tr Ž B⌺ . q ␮X B␮

2

2

Žsee Searle, 1971, Section 2.5 for expressions for the mean and variance of a quadratic form in normal variables.. EXAMPLE 12.9.1. Consider the linear model, y s X␤ q ⑀ , where X is n = p of rank p, ␤ is a vector of unknown parameters, and ⑀ is a random error vector. Under the null hypothesis of no serial correlation in the elements of ⑀ , we have ⑀ ; N Ž0, ␴ 2 I.. The corresponding Durbin-Watson Ž1950, 1951. test statistic is given by ds

⑀X PA 1 P⑀ ⑀X P⑀

,

where P s I I X(XX X) I1 XX , and A 1 is the matrix 1 y1 0 . . A1 s . 0 0 0

y1 2 y1 . . . 0 0 0

0 y1 2 .. .

0 0 y1 .. .

⭈⭈⭈ ⭈⭈⭈ ⭈⭈⭈ .. .

⭈⭈⭈ ⭈⭈⭈ ⭈⭈⭈

y1 0 0

2 y1 0

0 0 0 . . . y1 2 y1

0 0 0 . . . . 0 y1 1

548

APPROXIMATION OF INTEGRALS

Then, by Ž12.58., the Laplace approximation of E Ž d . is EL Ž d . s

E Ž ⑀X PA 1 P⑀ .

Ž 12.59 .

.

E Ž ⑀X P⑀ .

Durbin and Watson Ž1951. showed that d is distributed independently of its own denominator, so that the moments of the ratio d are the ratios of the corresponding moments of the numerator and denominator, that is, E Ž ⑀X PA 1 P⑀ .

EŽ d . s k

E Ž ⑀X P⑀ .

k

Ž 12.60 .

.

k

From Ž12.59. and Ž12.60. we note that the Laplace approximation for the mean, E Ž d ., is exact. For k G 2, Lieberman Ž1994. showed that EL Ž d k . EŽ d . k

s1qO

ž / 1

n

.

Thus, the relative error of approximating higher-order moments of d is O Ž1rn., regardless of the matrix X. 12.9.4. Laplace’s Approximation in Bayesian Statistics Suppose ŽKass, Tierney, and Kadane, 1991. that a data vector y s Ž y 1 , y 2 , . . . , yn .X has a distribution with the density function pŽy < ␪ ., where ␪ is an unknown parameter. Let LŽ ␪ . denote the corresponding likelihood function, which is proportional to pŽy < ␪ .. In Bayesian statistics, a prior density, ␲ Ž ␪ ., is assumed on ␪ , and inferences are based on the posterior density q Ž ␪ < y., which is proportional to LŽ ␪ .␲ Ž ␪ ., where the proportionality constant is determined by requiring that q Ž ␪ < y. integrate to one. For a given real-valued function g Ž ␪ ., its posterior expectation is given by

E gŽ␪ . y s

H g Ž ␪ . L Ž ␪ . ␲ Ž ␪ . d␪ H

L Ž ␪ . ␲ Ž ␪ . d␪

.

Ž 12.61 .

Tierney, Kass, and Kadane Ž1989. expressed the integrands in Ž12.61. as follows: g Ž ␪ . L Ž ␪ . ␲ Ž ␪ . s bN Ž ␪ . exp ynh N Ž ␪ . , L Ž ␪ . ␲ Ž ␪ . s bD Ž ␪ . exp ynh D Ž ␪ . ,

549

APPLICATIONS IN STATISTICS

where bN Ž ␪ . and bD Ž ␪ . are smooth functions that do not depend on n and h N Ž ␪ . and h D Ž ␪ . are constant-order functions of n, as n ™ ⬁. Formula Ž12.61. can then be written as

E gŽ␪ . y s

Hb

N

Hb

D

Ž ␪ . exp ynh N Ž ␪ . d␪ Ž ␪ . exp ynh D Ž ␪ . d␪

Ž 12.62 .

.

Applying Laplace’s approximation in Ž12.25. to the integrals in Ž12.62., we obtain, if n is large, E gŽ␪ . y f

hYD Ž ␪ˆD .

hYN Ž ␪ˆN .

1r2

bN Ž ␪ˆN . exp ynh N Ž ␪ˆN . bD Ž ␪ˆD . exp ynh D Ž ␪ˆD .

,

Ž 12.63 .

where ␪ˆN and ␪ˆD are the locations of the single maxima of yh N Ž ␪ . and yh D Ž ␪ ., respectively. In particular, if we choose h N Ž ␪ . s h D Ž ␪ . s yŽ1rn. logw LŽ ␪ .␲ Ž ␪ .x, bN Ž ␪ . s g Ž ␪ . and bD Ž ␪ . s 1, then Ž12.63. reduces to E g Ž ␪ . y f g Ž ␪ˆ. ,

Ž 12.64 .

where ␪ˆ is the point at which Ž1rn. logw LŽ ␪ .␲ Ž ␪ .x attains its maximum. Formula Ž12.64. provides a first-order approximation of E w g Ž ␪ . yx. This approximation is often called the modal approximation because ␪ˆ is the mode of the posterior density. A more accurate second-order approximation of E w g Ž ␪ . yx was given by Kass, Tierney, and Kadane Ž1991.. 12.9.5. Other Methods of Approximating Integrals in Statistics There are several major techniques and approaches available for the numerical approximation of integrals in statistics that are beyond the scope of this book. These techniques, which include the saddlepoint approximation and Marko® chain Monte Carlo, have received a great deal of attention in the statistical literature in recent years. The saddlepoint method is designed to approximate integrals of the Laplace type in which both the integrand and contour of integration are allowed to be complex valued. It is a powerful tool for obtaining accurate expressions for densities and distribution functions. A good introduction to the basic principles underlying this method was given by De Bruijn Ž1961, Chapter 5.. Daniels Ž1954. is credited with having introduced it in statistics in the context of approximating the density of a sample mean of independent and identically distributed random variables. Markov chain Monte Carlo ŽMCMC. is a general method for the simulation of stochastic processes having probability densities known up to a

550

APPROXIMATION OF INTEGRALS

constant of proportionality. It generally deals with high-dimensional statistical problems, and has come into prominence in statistical applications during the past several years. Although MCMC has potential applications in several areas of statistics, most attention to date has been focused on Bayesian applications. For a review of these techniques, see, for example, Geyer Ž1992., Evans and Swartz Ž1995., Goutis and Casella Ž1999., and Strawderman Ž2000..

FURTHER READING AND ANNOTATED BIBLIOGRAPHY Abramowitz, M., and I. A. Stegun, eds. Ž1972.. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Wiley, New York. ŽThis volume is an excellent source for a wide variety of numerical tables of mathematical functions. Chap. 25 gives zeros of Legendre, Hermite, and Laguerre polynomials along with their corresponding weight factors.. Copson, E. T. Ž1965.. Asymptotic Expansions. Cambridge University Press, London. ŽThe method of Laplace is discussed in Chap. 5.. Courant, R., and F. John Ž1965.. Introduction to Calculus and Analysis, Volume 1. Wiley, New York. ŽThe trapezoidal and Simpson’s methods are discussed in Chap. 6.. Daniels, H. Ž1954.. ‘‘Saddlepoint approximation in statistics.’’ Ann. Math. Statist., 25, 631᎐650. Davis, P. J., and P. Rabinowitz Ž1975.. Methods of Numerical Integration. Academic Press, New York. ŽThis book presents several useful numerical integration methods, including approximate integrations over finite or infinite intervals as well as integration in two or more dimensions.. De Bruijn, N. G. Ž1961.. Asymptotic Methods in Analysis, 2nd ed. North-Holland, Amsterdam. ŽChap. 4 covers the method of Laplace, and the saddlepoint method is the topic of Chap. 5.. Durbin, J., and G. S. Watson Ž1950.. ‘‘Testing for serial correlation in least squares regression, I.’’ Biometrika, 37, 409᎐428. Durbin, J., and G. S. Watson Ž1951.. ‘‘Testing for serial correlation in least squares regression, II.’’ Biometrika, 38, 159᎐178. Evans, M., and T. Swartz Ž1995.. ‘‘Methods for approximating integrals in statistics with special emphasis on Bayesian integration problems.’’ Statist. Sci., 10, 254᎐272. Flournoy, N., and R. K. Tsutakawa, eds. Ž1991.. Statistical Multiple Integration, Contemporary Mathematics 115. Amer. Math. Soc., Providence, Rhode Island. ŽThis volume contains the proceedings of an AMS᎐IMS᎐SIAM joint summer research conference on statistical multiple integration, which was held at Humboldt University, Arcata, California, June 17᎐23, 1989.. Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. ŽSection 18.3 of this book contains proofs associated with the method of Laplace. . Geyer, C. J. Ž1992.. ‘‘Practical Markov chain Monte Carlo.’’ Statist. Sci., 7, 473᎐511.

FURTHER READING AND ANNOTATED BIBLIOGRAPHY

551

Ghazal, G. A. Ž1994.. ‘‘Moments of the ratio of two dependent quadratic forms.’’ Statist. Prob. Letters, 20, 313᎐319. Gibaldi, M., and D. Perrier Ž1982.. Pharmacokinetics, 2nd ed. Dekker, New York. Goutis, C., and G. Casella Ž1999.. ‘‘Explaining the saddlepoint approximation.’’ Amer. Statist., 53, 216᎐224. Haber, S. Ž1970.. ‘‘Numerical evaluation of multiple integrals.’’ SIAM Re®., 12, 481᎐526. Kahaner, D. K. Ž1991.. ‘‘A survey of existing multidimensional quadrature routines.’’ In Statistical Multiple Integration, Contemporary Mathematics 115, N. Flournoy and R. K. Tsutakawa, eds. Amer. Math. Soc., Providence, pp. 9᎐22. Kass, R. E., L. Tierney, and J. B. Kadane Ž1991.. ‘‘Laplace’s method in Bayesian analysis.’’ In Statistical Multiple Integration, Contemporary Mathematics 115, N. Flournoy and R. K. Tsutakawa, eds. Amer. Math. Soc., Providence, pp. 89᎐99. Katz, D., and D. Z. D’Argenio Ž1983.. ‘‘Experimental design for estimating integrals by numerical quadrature, with applications to pharmacokinetic studies.’’ Biometrics, 39, 621᎐628. Krylov, V. I. Ž1962.. Approximate Calculation of Integrals. Macmillan, New York. ŽThis book considers only the problem of approximate integration of functions of a single variable. It was translated from the Russian by A. H. Stroud.. Lange, K. Ž1999.. Numerical Analysis for Statisticians. Springer, New York. ŽThis book contains a wide variety of topics on numerical analysis of potential interest to statisticians, including recent topics such as bootstrap calculations and the Markov chain Monte Carlo method.. Lieberman, O. Ž1994.. ‘‘A Laplace approximation to the moments of a ratio of quadratic forms.’’ Biometrika, 81, 681᎐690. Liu, Q., and D. A. Pierce Ž1994.. ‘‘A note on Gauss᎐Hermite quadrature.’’ Biometrika, 81, 624᎐629. Morin, D. Ž1992.. ‘‘Exact moments of ratios of quadratic forms.’’ Metron, 50, 59᎐78. Morland, T. Ž1998.. ‘‘Approximations to the normal distribution function.’’ Math. Gazette, 82, 431᎐437. Nonweiler, T. R. F. Ž1984.. Computational Mathematics. Ellis Horwood, Chichester, England. ŽNumerical quadrature is covered in Chap. 5.. Phillips, C., and B. Cornelius Ž1986.. Computational Numerical Methods. Ellis Horwood, Chichester, England. ŽNumerical integration is the subject of Chap. 6... Phillips, G. M., and P. J. Taylor Ž1973.. Theory and Applications of Numerical Analysis. Academic Press, New York. ŽGaussian quadrature is covered in Chap. 6... Piegorsch, W. W., and A. J. Bailer Ž1993.. ‘‘Minimum mean-square error quadrature.’’ J. Statist. Comput. Simul, 46, 217᎐234. Ralston, A., and P. Rabinowitz Ž1978.. A First Course in Numerical Analysis. McGraw-Hill, New York. ŽGaussian quadrature is covered in Chap. 4.. Reid, W. H., and S. J. Skates Ž1986.. ‘‘On the asymptotic approximation of integrals.’’ SIAM J. Appl. Math., 46, 351᎐358. Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts. Searle, S. R. Ž1971.. Linear Models. Wiley, New York.

552

APPROXIMATION OF INTEGRALS

Shoup, T. E. Ž1984.. Applied Numerical Methods for the Microcomputer. Prentice-Hall, Englewood Cliffs, New Jersey. Stark, P. A. Ž1970.. Introduction to Numerical Methods. Macmillan, London. Strawderman, R. L. Ž2000.. ‘‘Higher-order asymptotic approximation: Laplace, saddlepoint, and related methods.’’ J. Amer. Statist. Assoc., 95, 1358᎐1364. Stroud, A. H. Ž1971.. Approximate Calculation of Multiple Integrals. Prentice-Hall, Englewood Cliffs, New Jersey. Sutradhar, B. C., and R. F. Bartlett Ž1989.. ‘‘An approximation to the distribution of the ratio of two general quadratic forms with application to time series valued designs.’’ Comm. Statist. Theory Methods, 18, 1563᎐1588. Tierney, L., R. E. Kass, and J. B. Kadane Ž1989.. ‘‘Fully exponential Laplace approximations to expectations and variances of nonpositive functions.’’ J. Amer. Statist. Assoc., 84, 710᎐716. Wong, R. Ž1989.. Asymptotic Approximations of Integrals. Academic Press, New York. ŽThis is a useful reference book on the method of Laplace and Mellin transform techniques for multiple integrals. All results are accompanied by error bounds..

EXERCISES In Mathematics 12.1. Consider the integral In s

H1 log x dx. n

It is easy to show that In s n log n y n q 1. (a) Approximate In by using the trapezoidal method and the partition points x 0 s 1, x 1 s 2, . . . , x n s n, and verify that In f log Ž n! . y 12 log n. (b) Deduce from Ža. that n! and n nq1r2 eyn are of the same order of magnitude, which is essentially what is stated in Stirling’s formula Žsee Example 12.6.1. 12.2. Obtain an approximation of the integral H01dxrŽ1 q x . by Simpson’s method for the following values of n: 2, 4, 8, 16. Show that when n s 8, the error of approximation is less than or equal to 0.000002.

553

EXERCISES

12.3. Use Gauss᎐Legendre quadrature with n s 2 to approximate the value of the integral H0␲ r2 sin ␪ d␪ . Give an upper bound on the error of approximation using formula Ž12.16.. 12.4. (a) Show that ⬁

Hm e

yx 2

dx-

1 m

eym , 2

m ) 0.

w Hint: Use the inequality eyx - eym x for x) m.x (b) Find a value of m so that the upper bound in Ža. is smaller than 10y4 . 2 (c) Use part Žb. to find an approximation for H0⬁eyx dx correct to three decimal places. 2

12.5. Obtain an approximate value for the integral H0⬁ x Ž e x q eyx y 1.y1 dx using the Gauss᎐Laguerre quadrature. w Hint: Use the tables in Appendix C of the book by Krylov Ž1962, page 347. giving the zeros of Laguerre polynomials and the corresponding values of ␻ i .x 12.6. Consider the indefinite integral IŽ x. s

H0

dt

x

1qt2

s Arctan x.

(a) Make an appropriate change of variables to show that I Ž x . can be written as

Hy1 4 q x

IŽ x. s2 x

du

1

2

Ž u q 1.

2

(b) Use a five-point Gauss᎐Legendre quadrature to provide an approximation for I Ž x .. 12.7. Investigate the asymptotic behavior of the integral I Ž ␭. s

H0 Ž cos x . 1

␭

dx

as ␭ ™ ⬁. 12.8. (a) Use the Gauss-Laguerre quadrature to approximate the integral H0⬁ey6 x sin x dx using n s 1, 2, 3, 4, and compare the results with the true value of the integral.

554

APPROXIMATION OF INTEGRALS

(b) Use the Gauss-Hermite quadrature to approximate the integral ⬁ Hy⬁ x expŽy3 x 2 . dx using n s 1, 2, 3, 4, and compare the results with the true value of the integral. 12.9. Give an approximation to the double integral 1 H0 H0Ž1yx . Ž1 y x 2 1r2 1

.

2 2 1r2 1 y x2

dx 1 dx 2

by applying the Gauss᎐Legendre rule to formulas Ž12.32. and Ž12.33.. 12.10. Show that H0⬁Ž1 q x 2 .yn dx is asymptotically equal to n ™ ⬁.

1 2

Ž␲rn.1r2 as

In Statistics 12.11. Consider a sample, X 1 , X 2 , . . . , X n , of independent and indentically distributed random variables from the standard normal distribution. Suppose that n is odd. The sample median is the Ž m q 1.th order statistic XŽ mq1. , where m s Ž n y 1.r2. It is known that XŽ mq1. has the density function f Ž x. sn

ž

/

ny1 ⌽mŽ x. 1y⌽Ž x. m

m

␾Ž x. ,

where

␾Ž x. s

ž /

1

exp y

'2␲

x2 2

is the standard normal density function, and ⌽Ž x. s

Hy⬁␾ Ž t . dt x

Žsee Roussas, 1973, page 194.. Since the mean of XŽ mq1. is zero, the variance of XŽ mq1. is given by Var XŽ mq1. s n

ž

ny1 m

/H

⬁

y⬁

x2⌽mŽ x. 1y⌽Ž x.

m

␾ Ž x . dx.

Obtain an approximation for this variance using the Gauss᎐Hermite quadrature for n s 11 and varying numbers of quadrature points.

555

EXERCISES

12.12. ŽMorland, 1998.. Consider the density function 1

␾Ž x. s

'2␲

eyx

2

r2

for the standard normal distribution. Show that if x G 0, then the cumulative distribution function, ⌽Ž x. s

1

'2␲ Hy⬁e x

yt 2 r2

dt,

can be represented as the sum of the series ⌽Ž x. s

1

q

2

x

'2␲

1y

x2

q

6

y

40

x8

q

x4

3456

x6 336

y ⭈⭈⭈ q Ž y1 .

x2n

n

Ž 2 n q 1 . 2 n n!

q ⭈⭈⭈ .

w Note: By truncating this series, we can obtain a polynomial approximation of ⌽ Ž x .. For example, we have the following approximation of order 11: ⌽Ž x. f

1 2

q

x

'2␲

1y

x2 6

q

x4 40

y

x6 336

q

x8

y

3456

x 10 42240

.x

12.13. Use the result in Exercise 12.12 to show that there exist constants a, b,c, d, such that ⌽Ž x. f

1 2

q

x

'2␲

1 q ax 2 q bx 4 1 q cx 2 q dx 4

.

12.14. Apply the method of importance sampling to approximate the value of 2 the integral H02 e x dx using a sample of size n s 150 from the distribution whose density function is g Ž x . s 14 Ž1 q x ., 0 F x F 2. Compare your answer with the one you would get from applying formula Ž12.36..

556

APPROXIMATION OF INTEGRALS

12.15. Consider the density function

f Ž x. s

⌫

ž

nq1

'n␲ ⌫

/

ž /ž

2

n

1q

2

x2 n

/

y 12 Ž nq1 .

y⬁ - x- ⬁,

,

for a t-distribution with n degrees of freedom Žsee Exercise 6.28.. Let x F Ž x . s Hy⬁ f Ž t . dt be its cumulative distribution function. Show that for large n, FŽ x. f

1

'2␲ Hy⬁e x

yt 2 r2

dt,

which is the cumulative distribution function for the standard normal. w Hint: Apply Stirling’s formula.x

APPENDIX

Solutions to Selected Exercises

CHAPTER 1 1.3. A j C ; B implies A ; B. Hence, A s A l B. Thus, A l B ; C implies A ; C. It follows that A l C s ⭋. 1.4. (a) xg A ` B implies that xg A but f B, or xg B but f A; thus x g A j B y A l B. Vice versa, if x g A j B y A l B, then xg A ` B. (c) xg A l Ž B ` D . implies that x g A and xg B ` D, so that either xg A l B but f A l D, or xg A l D but f A l B, so that xg Ž A l B . ` Ž A l D .. Vice versa, if xg Ž A l B . ` Ž A l D ., then either xg A l B but f A l D, or xg A l D but f A l B, so that either x is in A and B but f D, or x is in A and D, but f B; thus xg A l Ž B ` D .. 1.5. It is obvious that ␳ is reflexive, symmetric, and transitive. Hence, it is an equivalence relation. If Ž m 0 , n 0 . is an element in A, then its equivalence class is the set of all pairs Ž m, n. in A such that mrm 0 s nrn 0 , that is, mrns m 0rn 0 . 1.6. The equivalence class of Ž1, 2. consists of all pairs Ž m, n. in A such that m y n s y1. 1.7. The first elements in all four pairs are distinct. n n 1.8. (a) If y g f ŽD is1 A i ., then y s f Ž x ., where xg D is1 A i . Hence, if n xg A i and f Ž x . g f Ž A i . for some i, then f Ž x . g D is1 f Ž A i .; thus n n f ŽD is1 A i . ; D is1 f Ž A i .. Vice versa, it is easy to show that n n D is1 f Ž A i . ; f ŽD is1 A i .. n (b) If y g f ŽF is1 . A i , then y s f Ž x ., where xg A i for all i; then n Ž . Ž . f x g f A i for all i; then f Ž x . g F is1 f Ž A i .. Equality holds if f is one-to-one.

557

558

SOLUTIONS TO SELECTED EXERCISES

1.11. Define f : Jq™ A such that f Ž n. s 2 n2 q 1. Then f is one-to-one and onto, so A is countable. 1.13. aq 'b s c q 'd ´ ay c s 'd y 'b . If as c, then bs d. If a/ c, then 'd y 'b is a nonzero rational number and 'd q 'b s Ž dy b.rŽ'd y 'b .. It follows that both 'd y 'b and 'd q 'b are rational numbers, and therefore 'b and 'd must be rational numbers. 1.14. Let g s infŽ A.. Then g F x for all x in A, so yg G yx, hence, yg is an upper bound of yA and is the least upper bound: if yg X is any other upper bound of yA, then yg X G yx, so xG g X , hence, g X is a lower bound of A, so g X F g, that is, yg X G yg, so yg s supŽyA.. 1.15. Suppose that bf A. Since b is the least upper bound of A, it must be a limit point of A: for any ⑀ ) 0, there exists an element ag A such that a) by ⑀ . Furthermore, a- b, since bf A. Hence, b is a limit point of A. But A is closed; hence, by Theorem 1.6.4, bg A. 1.17. Suppose that G is a basis for F , and let pg B. Then B s D␣ U␣ , where U␣ belongs to G. Hence, there is at least one U␣ such that pg U␣ ; B. Vice versa, if for each B g F and each pg B there is a U g G such that pg U ; B, then G must be a basis for F : for each pg B we can find a set Up g G such that pg Up ; B; then B s D
Up pg B4 , so G is a basis. 1.18. Let p be a limit point of A j B. Then p is a limit point of A or of B. In either case, pg A j B. Hence, by Theorem 1.6.4, A j B is a closed set. 1.19. Let
C␣ 4 be an open covering of B. Then B and
C␣ 4 form an open covering of A. Since A is compact, a finite subcollection of the latter covering covers A. Furthermore, since B does not cover B, the members of this finite covering that contain B are all in
C␣ 4 . Hence, B is compact. 1.20. No. Let Ž A, F . be a topological space such that A consists of two points a and b and F consists of A and the empty set ⭋. Then A is compact, and the point a is a compact subset; but it is not closed, since the complement of a, namely b, is not a member of F , and is therefore not open. 1.21. (a) Let w g A. Then X Ž w . F x- xq 3yn for all n, so w g F ⬁ns1 A n . Vice versa, if w g F ⬁ns1 A n , then X Ž w . - xq 3yn for all n. To show that X Ž w . F x: if X Ž w . ) x, then X Ž w . ) x q 3yn for some n, a contradiction. Therefore, X Ž w . F x, and w g A.

559

CHAPTER 1

(b) Let w g B. Then X Ž w . - x, so X Ž w . - xy 3yn for some n, hence, w g D ⬁ns1 Bn . Vice versa, if w g D ⬁ns1 Bn , then X Ž w . F xy 3yn for some n, so X Ž w . - x: if X Ž w . G x, then X Ž w . ) x y 3yn for all n, a contradiction. Therefore, w g B. 1.22. (a) P Ž X G 2. F ␭r2, since ␮ s ␭. (b) P Ž X G 2. s 1 y p Ž X - 2. s 1 y p Ž 0. y p Ž 1. s 1 y ey␭ Ž ␭ q 1 . . To show that 1 y ey␭ Ž ␭ q 1 . -

␭ 2

.

Let ␾ Ž ␭. s ␭r2 q ey␭ Ž ␭ q 1. y 1. Then i. ␾ Ž0. s 0, and ii. ␾ X Ž ␭. s 12 y ␭ ey␭ ) 0 for all ␭. From Ži. and Žii. we conclude that ␾ Ž ␭. ) 0 for all ␭ ) 0. 1.23. (a) pŽ X y ␮ G c. s P Ž X y ␮. G c2 2

F

␴2 c2

,

by Markov’s inequality and the fact that E Ž X y ␮ . 2 s ␴ 2 . (b) Use c s k ␴ in Ža.. (c ) P Ž X y ␮ - k␴ . s 1 y P Ž X y ␮ G k␴ . G1y

1 k2

.

1 1 x Ž1 y < x < . dxs 0, ␴ 2 s Hy1 x 2 Ž1 y < x < . dxs 16 . 1.24. ␮ s E Ž X . s Hy1 (a ) i. P Ž< X < G 12 . F ␴ 2r 14 s 23 . ii. P Ž< X < ) 13 . s P Ž< X < G 13 . F ␴ 2r 19 s 32 . (b)

P Ž X G 12 . s P Ž X G 12 . q P Ž X F y 12 . s 14 - 23 .

560

SOLUTIONS TO SELECTED EXERCISES

1.25. (a) True, since XŽ1. G x if and only if X i G x for all i. (b) True, since XŽ n. F x if and only if X i F x for all i. (c ) P Ž XŽ1. F x . s 1 y P Ž XŽ1. ) x . s1y 1yFŽ x.

n

.

(d) P Ž XŽ n. F x . s w F Ž x .x n. 1.26. P Ž2 F XŽ1. F 3. s P Ž XŽ1. F 3. y P Ž XŽ1. F 2.. Hence, P Ž 2 F XŽ1. F 3 . s 1 y 1 y F Ž 3 . s 1 y F Ž 2.

5

5

y 1 q 1 y F Ž 2.

5

y 1 y F Ž 3. . 5

But F Ž x . s H0x 2 ey2 t dts 1 y ey2 x. Hence, P Ž 2 F XŽ1. F 3 . s Ž ey4 . y Ž ey6 . 5

5

s ey20 y ey30 . CHAPTER 2 2.1. If m ) n, then the rank of the n = m matrix U s wu 1 : u 2 : ⭈⭈⭈ : u m x is less than or equal to n. Hence, the number of linearly independent columns of U is less than or equal to n, so the m columns of U must be linearly dependent. 2.2. If u 1 , u 2 , . . . , u n and v are linearly dependent, then v must belong to W, a contradiction. 2.5. Suppose that u 1 , u 2 , . . . , u n are linearly independent in U, and that Ý nis1 ␣ i T Žu i . s 0 for some constants ␣ 1 , ␣ 2 , . . . , ␣ n . Then T ŽÝ nis1 ␣ i u i . s 0. If T is one-to-one, then Ý nis1 ␣ i u i s 0, hence, ␣ i s 0 for all i, since the u i ’s are linearly independent. It follows that T Žu 1 ., T Žu 2 ., . . . , T Žu n . must also be linearly independent. Vice versa, let u g U such that T Žu. s 0. Let e 1 , e 2 , . . . , e n be a basis for U. Then u s Ý nis1␶ i e i for some constants ␶ 1 , ␶ 2 , . . . , ␶n . T Žu. s 0 ´ Ý nis1␶ i T Že i . s 0 ´ ␶ i s 0 for all i, since T Že 1 ., T Že 2 ., . . . , T Že n . are linearly independent. It follows that u s 0 and T is one-to-one. X . s Ý i Ý j a2i j . Hence, A s 0 if and only if 2.7. If A s Ž a i j ., then tr ŽAA X Ž . tr AA s 0.

561

CHAPTER 2

2.8. It is sufficient to show that Av s 0 if v XAv s 0. If v XAv s 0, then A1r2 v s 0, and hence Av s 0. w Note: A1r2 is defined as follows: since A is symmetric, it can be written as A s P⌳ P X by Theorem 2.3.10, where ⌳ is a diagonal matrix whose diagonal elements are the eigenvalues of A, and P is an orthogonal matrix. Furthermore, since A is positive semidefinite, its eigenvalues are greater than or equal to zero by Theorem 2.3.13. The matrix A1r2 is defined as P⌳1r2 P X , where the diagonal elements of ⌳1r2 are the square roots of the corresponding diagonal elements of ⌳ .x 2.9. By Theorem 2.3.15, there exists an orthogonal matrix P and diagonal matrices ⌳ 1 , ⌳ 2 such that A s P⌳ 1 P X , B s P⌳ 2 P X . The diagonal elements of ⌳ 1 and ⌳ 2 are nonnegative. Hence, AB s P⌳ 1 ⌳ 2 P X is positive semidefinite, since the diagonal elements of ⌳ 1 ⌳ 2 are nonnegative. 2.10. Let C s AB. Then CX s y BA. Since tr ŽAB. s tr ŽBA., we have tr ŽC. s tr Žy CX . s ytr ŽC. and thus tr ŽC. s 0. 2.11. Let B s A y AX . Then tr Ž BX B . s tr Ž AX y A . Ž A y AX . s tr Ž AX y A . A y tr Ž AX y A . AX s tr Ž AX y A . A y tr A Ž A y AX . s tr Ž AX y A . A y tr Ž A y AX . A s 2 tr Ž AX y A . A s 0, since AX y A is skew-symmetric. Hence, by Exercise 2.7, B s 0 and thus A s AX . 2.12. This follows easily from Theorem 2.3.9. 2.13. By Theorem 2.3.10, we can write A y ␭I n s P Ž ⌳ y ␭I n . P X , where P is an orthogonal matrix and ⌳ is diagonal with diagonal elements equal to the eigenvalues of A. The diagonal elements of ⌳ y ␭I n are the eigenvalues of A y ␭I n . Now, k diagonal elements of ⌳ y ␭I n are equal to zero, and the remaining n y k elements must be different from zero. Hence, A y ␭I n has rank n y k.

562

SOLUTIONS TO SELECTED EXERCISES

2.17. If AB s BA s B, then ŽA y B. 2 s A2 y AB y BA q B 2 s A y 2B q B s A y B. Vice versa, if A y B is idempotent, then A y B s ŽA y B. 2 s A2 y AB y BA q B 2 s A y AB y BA q B. Thus, AB q BA s 2B, so AB q ABA s 2AB and thus AB s ABA. We also have ABA q BA s 2BA,

hence,

BA s ABA.

It follows that AB s BA. We finally conclude that B s AB s BA. 2.18. If A is an n = n orthogonal matrix with determinant 1, then its eigenvalues are of the form e " i ␾ 1 , e " i ␾ 2 , . . . , e " i ␾ q , 1, where 1 is of multiplicity n y 2 q and none of the real numbers ␾ j is a multiple of 2␲ Ž j s 1, 2, . . . , q . Žthose that are odd multiples of ␲ give an even number of eigenvalues equal to y1.. 2.19. Using Theorem 2.3.10, it is easy to show that emin Ž A . I n F A F emax Ž A . I n , where the inequality on the right means that emax ŽA.I n y A is positive semidefinite, and the one on the left means that A y emin ŽA.I n is positive semidefinite. It follows that emin Ž A . LX L F LXALF emax Ž A . LX L Hence, by Theorem 2.3.19Ž1., emin Ž A . tr Ž LX L . F tr Ž LXAL. F emax Ž A . tr Ž LX L . . X 2.20. (a) We have that A G emin ŽA.I n by Theorem 2.3.10. Hence, LALG X X X emin ŽA.L L, and therefore, emin ŽLAL. G emin ŽA. emin ŽL L. by Theorem 2.3.18 and the fact that emin ŽA. G 0 and LX L is nonnegative definite. (b) This is similar to Ža..

2.21. We have that emin Ž B . A F A1r2 BA1r2 F emax Ž B . A, since A is nonnegative definite. Hence, emin Ž B . tr Ž A . F tr Ž A1r2 BA1r2 . F emax Ž B . tr Ž A . .

563

CHAPTER 2

The result follows by noting that tr Ž AB . s tr Ž A1r2 BA1r2 . . 2.23. (a) A s I n y Ž1rn.J n , where J n is the matrix of ones of order n = n. A2 s A, since J n2 s nJ n . The rank of A is the same as its trace, which is equal to n y 1. 2 (b) Ž n y 1. s 2r␴ 2 s Ž1r␴ 2 .y XAy; ␹ny1 , since Ž1r␴ 2 .AŽ ␴ 2 I n . is idempotent of rank n y 1, and the noncentrality parameter is zero. (c ) 1 X Cov Ž y, Ay. s Cov 1 y, Ay n n

ž

s s

1 n

/

1Xn Ž ␴ 2 I n . A

␴2 n

1Xn A

s 0X . Since y is normally distributed, both y and Ay, being linear transformations of y, are also normally distributed. Hence, they must be independent, since they are uncorrelated. We can similarly show that y and A1r2 y are uncorrelated, and hence independent, by the fact that 1Xn A s 0X and thus 1Xn A1 n s 0, which means 1Xn A1r2 s 0X . It follows that y and y XA1r2 A1r2 y s y XAy are independent. 2.24. (a) The model is written as y s Xg q ⑀ , a where X s w1 N : [is1 1 n i x, [ denotes the direct sum of matrices, g s Ž ␮ , ␣ 1 , ␣ 2 , . . . , ␣ a .X , and N s Ý ais1 n i . Now, ␮ q ␣ i is estimable, since it can be written as aXi g, where aXi is a row of X, i s 1, 2, . . . , a. It follows that ␣ i y ␣ iX s ŽaXi y aXiX .g is estimable, since the vector aXi y aXiX belongs to the row space of X. (b) Suppose that ␮ is estimable; then it can be written as

a

␮s

Ý ␶i Ž ␮ q ␣i . ,

is1

where ␶ 1 , ␶ 2 , . . . , ␶a are constants, since ␮ q ␣ 1 , ␮ q ␣ 2 , . . . , ␮ q ␣ a form a basic set of estimable linear functions. We must therefore have Ý aisl ␶ i s 1, and ␶ i s 0 for all i, a contradiction. Therefore, ␮ is nonestimable.

564

SOLUTIONS TO SELECTED EXERCISES

2.25. (a) XŽXX X.y XX XŽXX X.y XX s XŽXX X.y XX by Theorem 2.3.3Ž2.. (b) E Ž l X y. s ␭X ␤ ´ l X X␤ s ␭X ␤. Hence, ␭X s l X X. Now, y

y

ˆ . s ␭X Ž XX X . XX X Ž XX X . ␭ ␴ 2 Var Ž ␭X ␤ X

y

X

y

y

s l X Ž XX X . XX X Ž XX X . XX l ␴ 2 s l X Ž XX X . XX l ␴ 2 ,

by Theorem 2.3.3 Ž 2 .

X

F l l␴ 2, since I n y XŽXX X.y XX is positive semidefinite. The result follows X X from the last inequality, since VarŽ l y. s l l ␴ 2 ,

ˆ s P⌳y1 PX XX y, so ⌳ PX ␤ ˆ s PX XX y. 2.26. ␤U s PŽ ⌳ q kI p .y1 P X XX y. But ␤ Hence, ␤U s P Ž ⌳ q kI p .

y1

ˆ ⌳ PX ␤

ˆ s PDP X ␤. 2.27. y1 sup ␳ 2 s sup Ž ␯X Cy1 1 AC 2 ␶ .

x, y

2

␯, ␶

½ ½

␶

½

␶

y1 s sup sup Ž ␯X Cy1 1 AC 2 ␶ .

␯

␶

s sup sup Ž bX ␶ . ␯

2

5

s sup sup Ž ␶X bbX ␶ . ␯

2

s sup
emax Ž bbX . 4 , ␯

5

y1 Ž bX s ␯X Cy1 1 AC 2 .

5 by applying Ž 2.9 .

s sup
emax Ž bX b . 4 ␯

y2 X y1 s sup
␯X Cy1 1 AC 2 A C 1 ␯ 4

␯

y2 X y1 s emax Ž Cy1 1 AC 2 A C 1 . , y2 X s emax Ž Cy2 1 AC 2 A . y1 X s emax Ž By1 1 AB 2 A . .

by Ž 2.9 .

565

CHAPTER 3

CHAPTER 3 3.1. (a) lim

x5 y1

x™1

xy 1

s lim Ž 1 q xq x 2 q x 3 q x 4 . x™1

s 5. (b) xsin Ž 1rx . F x ´ lim x ™ 0 x sin Ž1rx . s 0. (c) The limit does not exist, since the function is equal to 1 except when sin Ž1rx . s 0, that is, when x s .1r␲ , . 1r2␲ , . . . , . 1rn␲ , . . . . For such values, the function takes the form 0r0, and is therefore undefined. To have a limit at xs 0, the function must be defined at all points in a deleted neighborhood of x s 0. (d) lim x ™ 0y f Ž x . s 12 and lim x ™ 0q f Ž x . s 1. Therefore, the function f Ž x . does not have a limit as x™ 0. 3.2. (a) To show that Žtan x 3 .rx 2 ™ 0 as x ™ 0: Žtan x 3 .rx 2 s Ž1rcos x 3 .Žsin x 3 .rx 2 . But sin x 3 F x 3 . Hence, Ž tan x 3 . rx 2 F < x < r < cos x 3 < ™ 0 as x™ 0. (b) xr 'x ™ 0 as x™ 0. (c) O Ž1. is bounded as x™ ⬁. Therefore, O Ž1.rx™ 0 as x™ ⬁, so O Ž1. s oŽ x . as x™ ⬁. (d) f Ž x . g Ž x . s xq o Ž x 2 . s

1 x

qx O

ž / 1 x

1 x

2

q

qO 1 x

2

ž / 1 x

oŽ x 2 . q oŽ x 2 . O

ž / 1 x

.

Now, by definition, xO Ž 1rx . is bounded by a positive constant as x™ 0, and oŽ x 2 .rx 2 ™ 0 as x™ 0. Hence, oŽ x 2 .rx 2 is bounded, that is, O Ž1.. Furthermore, oŽ x 2 . O Ž1rx . s x w oŽ x 2 .rx 2 x xO Ž1rx ., which goes to zero as x™ 0, is also bounded. It follows that f Ž x. gŽ x. s

1 x

q O Ž 1. .

3.3. (a) f Ž0. s 0 and lim x ™ 0 f Ž x . s 0. Hence, f Ž x . is continuous at xs 0. It is also continuous at all other values of x. (b) The function f Ž x . is defined on w1, 2x, but lim x ™ 2y f Ž x . s ⬁, so f Ž x . is not continuous at xs 2.

566

SOLUTIONS TO SELECTED EXERCISES

(c) If n is odd, then f Ž x . is continuous everywhere, except at x s 0. If n is even, f Ž x . will be continuous only when x) 0. Ž m and n must be expressed in their lowest terms so that the only common divisor is 1.. (d) f Ž x . is continuous for x/ 1. 3.6. f Ž x. s

½

y3, 0,

x/ 0, xs 0.

Thus f Ž x . is continuous everywhere except at xs 0. 3.7. lim x ™ 0y f Ž x . s 2 and lim x ™ 0q f Ž x . s 0. Therefore, f Ž x . cannot be made continuous at xs 0. 3.8. Letting as bs 0 in f Ž aq b . s f Ž a. q f Ž b ., we conclude that f Ž0. s 0. Now, for any x 1 , x 2 g R, f Ž x1 . s f Ž x 2 . q f Ž x1 y x 2 . . Let z s x 1 y x 2 . Then f Ž x1 . y f Ž x 2 . s f Ž z . s f Ž z . y f Ž 0. . Since f Ž x . is continuous at xs 0, for a given ⑀ ) 0 there exists a ␦ ) 0 such that f Ž z . y f Ž 0 . - ⑀ if z - ␦ . Hence, f Ž x 1 . y f Ž x 2 . - ⑀ for all x 1 , x 2 such that x 1 y x 2 - ␦ . Thus f Ž x . is uniformly continuous everywhere in R. 3.9. lim x ™ 1y f Ž x . s lim x ™ 1q f Ž x . s f Ž1. s 1, so f Ž x . is continuous at xs 1. It is also continuous everywhere else on w0, 2x, which is closed and bounded. Therefore, it is uniformly continuous by Theorem 3.4.6. 3.10. cos x 1 y cos x 2 s 2sin F 2 sin

ž ž

x 2 y x1

sin

2 x 2 y x1

F x1 y x 2

2

/ ž /

x 2 q x1 2

/

for all x 1 , x 2 in R.

Thus, for a given ⑀ ) 0, there exists a ␦ ) 0 Žnamely, ␦ - ⑀ . such that cos x 1 y cos x 2 - ⑀ whenever x 1 y x 2 - ␦ .

567

CHAPTER 3

3.13. f Ž x . s 0 if x is a rational number in w a, b x. If x is an irrational number, then it is a limit of a sequence
yn4⬁ns1 of rational numbers Žany neighborhood of x contains infinitely many rationals.. Hence,

ž

/

f Ž x . s f lim yn s lim f Ž yn . s 0, n™⬁

n™⬁

since f Ž x . is a continuous function. Thus f Ž x . s 0 for every x in w a, b x. 3.14. f Ž x . can be written as

½

3 y 2 x, f Ž x . s 5, 3 q 2 x,

xF y1, y1 - x- 1, xG 1.

Hence, f Ž x . has a unique inverse for xF y1 and for xG 1. 3.15. The inverse function is xs fy1 Ž y . s

½

y F 1,

y, 1 2

Ž y q 1. ,

y ) 1.

' '

3.16. (a) The inverse of f Ž x . is fy1 Ž y . s 2 y yr2 . (b) The inverse of f Ž x . is fy1 Ž y . s 2 q yr2 . 3.17. By Theorem 3.6.1, n

Ý f Ž ai . y f Ž bi .

n

FK

is1

Ý

a i y bi

is1

-K␦ . Choosing ␦ such that ␦ - ⑀rK, we get n

Ý f Ž ai . y f Ž bi .

-⑀ ,

is1

for any given ⑀ ) 0. 3.18. This inequality can be proved by using mathematical induction: it is obviously true for n s 1. Suppose that it is true for n s m. To show that it is true for n s m q 1. For n s m, we have f

ž

Ým is1 a i x i Am

/

F

Ým is1 a i f Ž x i . Am

,

568

SOLUTIONS TO SELECTED EXERCISES

where A m s Ý m is1 a i . Let Ým is1 a i x i

bm s

Am

.

Then f

ž

Ý mq1 is1 a i x i A mq1

/ ž sf

F

A m bm q a mq1 x mq1 A mq1

Am A mq1

a mq1

f Ž bm . q

A mq1

/ f Ž x mq1 . .

Ž . But f Ž bm . F Ž1rA m .Ý m is1 a i f x i . Hence, f

ž

Ý mq1 is1 a i x i A mq1

/

F

Ým is1 a i f Ž x i . q a mq1 f Ž x mq1 .

s

Ý mq1 is1 a i f Ž x i .

A mq1

A mq1

.

3.19. Let a be a limit point of S. There exists a sequence
a n4⬁ns1 in S such that lim n™⬁ a n s a Žif S is finite, then S is closed already.. Hence, f Ž a. s f Žlim n™⬁ a n . s lim n™⬁ f Ž a n . s 0. It follows that ag S, and S is therefore a closed set. 3.20. Let g Ž x . s expw f Ž x .x. We have that f ␭ x1 q Ž 1 y ␭. x 2 F ␭ f Ž x1 . q Ž 1 y ␭. f Ž x 2 . for all x 1 , x 2 in D, 0 F ␭ F 1. Hence, g ␭ x 1 q Ž 1 y ␭ . x 2 F exp ␭ f Ž x 1 . q Ž 1 y ␭ . f Ž x 2 . F ␭ g Ž x1 . q Ž 1 y ␭. g Ž x 2 . , since the function e x is convex. Hence, g Ž x . is convex on D. 3.22. (a) We have that E Ž X . G E Ž X . . Let X ; N Ž ␮ , ␴ 2 .. Then EŽ X . s

1

'2␲␴ 2

⬁

1

Hy⬁exp y 2 ␴

2

2 Ž xy ␮ . x dx.

569

CHAPTER 3

Therefore, EŽ X . F

1

'2␲␴

sEŽ X

1

⬁

Hy⬁exp y 2 ␴

2

2

Ž xy ␮ .

2

x dx

..

(b) We have that E Ž eyX . G ey␮ , where ␮ s E Ž X ., since eyx is a convex function. The density function of the exponential distribution with mean ␮ is 1

gŽ x. s

␮

eyx r ␮ ,

0 - x - ⬁.

Hence, E Ž eyX . s s

1

⬁

He ␮ 0 1

yx r ␮ yx

e

1

␮ 1 q 1r␮

s

dx 1

␮q1

.

But e ␮s1q␮q

1 2!

␮2 q ⭈⭈⭈ q

1 n!

␮ n q ⭈⭈⭈

G 1 q ␮. It follows that ey␮ F

1

␮q1

and thus E Ž eyX . G ey␮ . 3.23. P Ž X n2 G ⑀ . s P Ž X n G ⑀ 1r2 . ™ 0 as n ™ ⬁, since X n converges in probability to zero. 3.24.

␴ 2 s E Ž Xy␮. sE GE2

Xy␮

2

2

Xy␮ ,

hence,

␴GE Xy␮ .

570

SOLUTIONS TO SELECTED EXERCISES

CHAPTER 4 4.1. lim h™0

f Ž h . y f Ž yh . 2h

f Ž h . y f Ž 0.

s lim

2h

h™0 X

s

f Ž 0.

q

2 f Ž 0.

f Ž 0 . y f Ž yh .

h™0

lim

2

X

s

1

q lim

2h

f Ž yh . y f Ž 0 .

Ž yh .

h™0 X

q

2

f Ž 0. 2

s f X Ž 0. .

The converse is not true: let f Ž x . s < x < . Then f Ž h . y f Ž yh . 2h

s 0,

and its limit is zero. But f X Ž0. does not exist. 4.3. For a given ⑀ ) 0, there exists a ␦ ) 0 such that for any xg N␦ Ž x 0 ., x/ x 0 , f Ž x . y f Ž x0 . xy x 0

y f X Ž x0 . - ⑀ .

Hence, f Ž x . y f Ž x0 . xy x 0

- f X Ž x0 . q ⑀

and thus f Ž x . y f Ž x0 . - A x y x0 , where A s f X Ž x 0 . q ⑀ . 4.4. g Ž x . s f Ž xq 1. y f Ž x . s f X Ž ␰ ., where ␰ is between x and xq 1. As x™ ⬁, we have ␰ ™ ⬁ and f X Ž ␰ . ™ 0, hence, g Ž x . ™ 0 as x ™ ⬁. 4.5. We have that f X Ž1. s lim x ™ 1q Ž x 3 y 2 xq 1.rŽ x y 1. s lim x ™ 1y Ž ax 2 y bxq 1 q 1.rŽ xy 1., and thus 1 s 2 ay b. Furthermore, since f Ž x . is continuous at xs 1, we must have ay bq 1 s y1. It follows that as 3, bs 5, and

½

2 f X Ž x . s 3 x y 2, 6 xy 5,

which is continuous everywhere.

x G 1, x - 1,

571

CHAPTER 4

4.6. Let x) 0. (a) Taylor’s theorem gives f Ž xq 2 h . s f Ž x . q 2 hf X Ž x . q

Ž 2 h. 2!

2

fYŽ ␰ .,

where x- ␰ - xq 2 h, and h ) 0. Hence, fXŽ x. s

1 2h

f Ž xq 2 h . y f Ž x . y hf Y Ž ␰ . ,

so that f X Ž x . F m 0rh q hm 2 . (b) Since m1 is the least upper bound of f X Ž x . , m1 F

m0 h

q hm 2 .

Equivalently, m 2 h 2 y m1 h q m 0 G 0 This inequality is valid for all h ) 0 provided that the discriminant ⌬ s m12 y 4 m 0 m 2 is less than or equal to zero, that is, m12 F 4 m 0 m 2 . 4.7. No, unless f X Ž x . is continuous at x 0 . For example, the function xq 1, f Ž x . s 0, xy 1,

x) 0, xs 0, x- 0,

does not have a derivative at xs 0, but lim x ™ 0 f X Ž x . s 1. 4.8. DX Ž y j . s lim

a™y j

s lim

a™y j

y j y a q Ý i/ j yi y a y Ý nis1 yi y y j ay y j y j y a q Ý i/ j yi y a y Ý i/ j yi y y j ay y j

,

which does not exist, since lim a™ y j y j y a rŽ ay y j . does not exist.

572

SOLUTIONS TO SELECTED EXERCISES

4.9. d

f Ž x.

dx

x

s

x f XŽ x. yf Ž x. x2

.

x f X Ž x . y f Ž x . ™ 0 as x™ 0. Hence, by l’Hospital’s rule, lim x™0

d

f Ž x.

dx

x

s lim

f X Ž x. qx f Y Ž x. yf X Ž x. 2x

x™0

s

1 2

f Y Ž 0. .

4.10. Let x 1 , x 2 g R. Then, f Ž x1 . y f Ž x 2 . s Ž x1 y x 2 . f X Ž ␰ . , here ␰ is between x 1 and x 2 . Since f X Ž x . is bounded for all x, f X Ž ␰ . - M for some positive constant M. Hence, for a given ⑀ ) 0, there exists a ␦ ) 0, where M␦ - ⑀ , such that f Ž x 1 . y f Ž x 2 . - ⑀ if x 1 y x 2 - ␦ . Thus f Ž x . is uniformly continuous on R. 4.11. f X Ž x . s 1 q cg X Ž x ., and cg X Ž x . - cM. Choose c such that cM- 12 . In this case, cg X Ž x . - 12 , so y 12 - cg X Ž x . - 21 , and, 1 2

- f X Ž x . - 23 .

Hence, f X Ž x . is positive and f Ž x . is therefore strictly monotone increasing, thus f Ž x . is one-to-one. 4.12. It is sufficient to show that g X Ž x . G 0 on Ž0, ⬁.: X

g Ž x. s

x f X Ž x. yf Ž x. x2

,

x) 0.

By the mean value theorem, f Ž x . s f Ž 0. q x f X Ž c . , sx f X Ž c. .

0-c-x

573

CHAPTER 4

Hence, gX Ž x . s

x f X Ž x . yx f X Ž c. x2

s

f X Ž x . yf X Ž c. x

x) 0.

,

Since f X Ž x . is monotone increasing, we have for all x ) 0, f X Ž x . Gf X Ž c.

gX Ž x . G 0

and thus

4.14. Let y s Ž1 q 1rx . x. Then,

ž

log y s x log 1 q

1 x

/

s

log Ž 1 q 1rx . 1rx

.

Applying l’Hospital’s rule, we get y lim log y s lim

x™⬁

1 x2

x™⬁

ž

1q

y

1 x

1

/

y1

s 1.

x2

Therefore, lim x ™⬁ y s e. 4.15. (a) lim x ™ 0q f Ž x . s 1, where f Ž x . s Žsin x . x. (b) lim x ™ 0q g Ž x . s 0, where g Ž x . s ey1r xrx. 4.16. Let f Ž x . s w1 q axq oŽ x .x1r x , and let y s axq oŽ x .. Then y s x w aq oŽ1.x, and w1 q axq oŽ x .x1r x s Ž1 q y . a r y Ž1 q y . oŽ1.r y. Now, as x™ 0, we have y ™ 0, Ž1 q y . a r y ™ e a, and Ž1 q y . oŽ1.r y ™ 1, since o Ž 1. y

log Ž 1 q y . ™ 0

as y ™ 0.

It follows that f Ž x . ™ e a. 4.17. No, because both f X Ž x . and g X Ž x . vanish at x s 0.541 in Ž0, 1.. 4.18. Let g Ž x . s f Ž x . y ␥ Ž xy a.. Then gX Ž x . s f X Ž x . y ␥ , g X Ž a . s f X Ž a . y ␥ - 0, g X Ž b . s f X Ž b . y ␥ ) 0.

574

SOLUTIONS TO SELECTED EXERCISES

The function g Ž x . is continuous on w a, b x. Therefore, it must achieve its absolute minimum at some point ␰ in w a, b x. This point cannot be a or b, since g X Ž a. - 0 and g X Ž b . ) 0. Hence, a- ␰ - b, and g X Ž ␰ . s 0, so f XŽ ␰ . s ␥. 4.19. Define g Ž x . s f Ž x . y ␶ Ž xy a., where n

␶s

Ý ␭i f X Ž x i . .

is1

There exist c1 , c 2 such that max f X Ž x i . s f X Ž c 2 . ,

a- c 2 - b,

min f X Ž x i . s f X Ž c1 . ,

a- c1 - b.

i

i

If f X Ž x 1 . s f X Ž x 2 . s ⭈⭈⭈ s f X Ž x n ., then the result is obviously true. Let us therefore assume that these n derivatives are not all equal. In this case, f X Ž c1 . - ␶ - f X Ž c 2 . . Apply now the result in Exercise 4.18 to conclude that there exists a point c between c1 and c 2 such that f X Ž c . s ␶ . 4.20. We have that n

Ý

n

Ý f X Ž c i . Ž yi y x i . ,

f Ž yi . y f Ž x i . s

is1

is1

where c i is between x i and yi Ž i s 1, 2, . . . , n.. Using Exercise 4.19, there exists a point c in Ž a, b . such that Ý nis1 Ž yi y x i . f X Ž c i . Ý nis1 Ž yi y x i .

sf X Ž c. .

Hence, n

Ý

n

f Ž yi . y f Ž x i . s f X Ž c .

is1

4.21. logŽ1 q x . s Ý⬁ns1 Žy1. ny1 x nrn,

Ý Ž yi y x i . .

is1

x - 1.

4.23. f Ž x . has an absolute minimum at xs 13 .

575

CHAPTER 4

4.24. The function f Ž x . is bounded if x 2 q axq b/ 0 for all x in wy1, 1x. Let ⌬ s a2 y 4 b. If ⌬ - 0, then x 2 q axq b) 0 for all x. The denominator has an absolute minimum at xs yar2. Thus, if y1 F yar2F 1, then f Ž x . will have an absolute maximum at x s yar2. Otherwise, f Ž x . attains its absolute maximum at xs y1 or xs 1. If ⌬ s 0, then f Ž x. s

1

ž

xq

a 2

/

2

.

In this case, the point yar2 must fall outside wy1, 1x, and the absolute maximum of f Ž x . is attained at xs y1 or xs 1. Finally, if ⌬ ) 0, then f Ž x. s

1

Ž xy x 1 . Ž xy x 2 .

,

where x 1 s 12 Žyay '⌬ ., x 2 s 12 Žyaq '⌬ .. Both x 1 and x 2 must fall outside wy1, 1x. In this case, the point xs yar2 is equal to 12 Ž x 1 q x 2 ., which falls outside wy1, 1x. Thus f Ž x . attains its absolute maximum at xs y1 or xs 1. 4.25. Let H Ž y . denote the cumulative distribution function of Gy1 w F Ž Y .x. Then H Ž y . s P
Gy1 F Ž Y . F y 4 sP F Ž Y . FGŽ y. s P
Y F Fy1 G Ž y . s F
Fy1 G Ž y .

4

4

s GŽ y. . 4.26. Let g Ž w . be the density function of W. Then g Ž w . s 2 weyw , w G 0. 2

4.27. (a) Let g Ž w . be the density function of W. Then g Ž w. s

1 3w

2r3

'0.08␲

exp y

1 0.08

2

Ž w 1r3 y 1 . ,

w / 0.

(b) Exact mean is E ŽW . s 1.12. Exact variance is VarŽW . s 0.42. (c) E Ž w . f 1, VarŽ w . f 0.36.

576

SOLUTIONS TO SELECTED EXERCISES

4.28. Let G Ž y . be the cumulative distribution function of Y. Then GŽ y. sP Ž YFy. sP Ž Z2Fy. s P Ž Z F y 1r2 . s P Ž yy 1r2 F Z F y 1r2 . s 2 F Ž y 1r2 . y 1, where F Ž⭈. is the cumulative distribution function of Z. Thus the density function of Y is g Ž y . s 1r 2␲ y eyy r2 , y ) 0. This represents the density function of a chi-squared distribution with one degree of freedom.

'

4.29. (a) failure rate s

F Ž x q h. y F Ž x . 1yFŽ x.

.

(b) Hazard rate s s s

dF Ž x . rdx 1yFŽ x. 1 yx r ␴

e

1

␴

ž

1

␴

eyx r ␴

.

(c) If dF Ž x . rdx 1yFŽ x.

s c,

then ylog 1 y F Ž x . s cxq c1 , hence, 1 y F Ž x . s c 2 eyc x . Since F Ž0. s 0, c 2 s 1. Therefore, F Ž x . s 1 y eyc x .

/

577

CHAPTER 5

4.30. P Ž Yn s r . s s

ž /ž . Ž . ž

n Ž n y 1 . ⭈⭈⭈ Ž n y r q 1 .

␭t

r!

n

n Ž n y 1 . ⭈⭈⭈ Ž n y r q 1 nr

r

1y

r

␭t

r!

1y

␭t n

␭t n

/ / ž nyr

yr

1y

␭t n

/

n

.

As n ™ ⬁, the first r factors on the right tend to 1, the next factor is fixed, the next tends to 1, and the last factor tends to ey␭ t. Hence, lim P Ž Yn s r . s

ey␭ t Ž ␭ t . r!

n™⬁

r

.

CHAPTER 5 5.1. (a) The sequence
bn4⬁ns1 is monotone increasing, since max
a1 , a2 , . . . , a n 4 F max
a1 ,a2 , . . . , a nq1 4 . It is also bounded. Therefore, it is convergent by Theorem 5.1.2. Its limit is sup nG 1 a n , since a n F bn F sup nG 1 a n for n G 1. (b) Let d i s log a i y log c, i s 1, 2, . . . . Then log c n s

1 n

n

1

n

Ý log ai s log c q n Ý

is1

di .

is1

To show that Ž1rn.Ý nis1 d i ™ 0 as n ™ ⬁: We have that d i ™ 0 as i ™ ⬁. Therefore, for a given ⑀ ) 0, there exists a positive integer N1 such that d i - ⑀r2 if i ) N1. Thus, for n ) N1 , 1 n

n

Ý

di F

is1

-

1 n 1 n

N1

Ý

di q

is1 N1

Ý

di q

is1

⑀ 2n

⑀ 2

Ž n y N1 .

.

Furthermore, there exists a positive integer N2 such that 1 n

N1

Ý

is1

di -

⑀ 2

,

if n ) N2 . Hence, if n ) maxŽ N1 , N2 ., Ž 1rn . Ý nis1 d i - ⑀ , which implies that Ž1rn.Ý nis1 d i ™ 0, that is, log c n ™ log c, and c n ™ c as n ™ ⬁.

578

SOLUTIONS TO SELECTED EXERCISES

5.2. Let a and b be the limits of
a n4⬁ns1 and
bn4⬁ns1 , respectively. These limits exist because the two sequences are of the Cauchy type. To show that d n ™ d, where ds < ay b < : dn y d s

a n y bn y ay b

F Ž a n y bn . y Ž ay b . F a n y a q bn y b . It is now obvious that d n ™ d as a n ™ a and bn ™ b. 5.3. Suppose that a n ™ c. Then, for a given ⑀ ) 0, there exists a positive integer N such that a n y c - ⑀ if n ) N. Let bn s a k n be the nth term of a subsequence. Since k n G n, we have bn y c - ⑀ if n ) N. Hence, bn ™ c. Vice versa, if every subsequence converges to c, then a n ™ c, since
a n4⬁ns1 is a subsequence of itself. 5.4. If E is not bounded, then there exists a subsequence
bn4⬁ns1 such that bn ™ ⬁, where bn s a k n, k 1 - k 2 - ⭈⭈⭈ - k n - ⭈⭈⭈ . This is a contradiction, since
a n4⬁ns1 is bounded. 5.5. (a) For a given ⑀ ) 0, there exists a positive integer N such that a n y c - ⑀ if n ) N. Thus, there is a positive constant M such that a n y c - M for all n. Now, Ý nis1 ␣ i a i Ý nis1 ␣ i

yc s s

F

F

1 Ý nis1 ␣ i

n

Ý ␣ i Ž ai y c .

is1 N

1 Ý nis1 ␣ i 1 Ý nis1 ␣ i M Ý nis1 ␣ i

-M

Ý ␣ i Ž ai y c .

q

is1 N

Ý ␣i

ai y c q

is1 N

Ý ␣ i q Ýn

is1

N Ý is1 ␣i

Ý nis1 ␣ i

⑀

is1

Ý nis1 ␣ i

Ý nis1 ␣ i

␣i

isNq1

Hence, as n ™ ⬁, Ý nis1 ␣ i

y c ™ 0.

␣ i Ž ai y c .

isNq1

Ý

isNq1

n

Ý

Ý

n

1

q⑀ .

Ý nis1 ␣ i a i

n

1

␣i

␣ i ai y c

579

CHAPTER 5

(b) Let a n s Žy1. n. Then,
a n4⬁ns1 does not converge. But Ž1rn.Ý nis1 a i is equal to zero if n is even, and is equal to y1rn if n is odd. Hence, Ž1rn.Ý nis1 a i goes to zero as n ™ ⬁. 5.6. For a given ⑀ ) 0, there exists a positive integer N such that a nq1 an

yb -⑀

if n ) N. Since b- 1, we can choose ⑀ so that b q ⑀ - 1. Then a nq1 an

- bq ⑀ - 1,

n ) N.

Hence, for n G N q 1, a Nq2 - a Nq1 Ž b q ⑀ . , a Nq3 - a Nq2 Ž bq ⑀ . - a Nq1 Ž bq ⑀ . , . . . a Nq1 n nyNy1 a n - a Nq1 Ž b q ⑀ . s bq ⑀ . . Nq1 Ž Ž bq ⑀ . 2

Letting c s a Nq1 rŽ bq ⑀ . Nq1 , r s bq ⑀ , we get a n - cr n, where 0 r - 1, for n G N q 1. 5.7. We first note that for each n, a n ) 0, which can be proved by induction. Let us consider two cases. 1. b) 1. In this case, we can show that the sequence is Ži. bounded from above, and Žii. monotone increasing: i. The sequence is bounded from above by 'b , that is, a n - 'b : a2n - b is true for n s 1 because a1 s 1 - b. Suppose now that a2n - b; to show that a2nq1 - b:

by a2nq1

s by

a2n Ž 3bq a2n .

Ž 3a2n q b .

2

Ž by a2n . s ) 0. 2 Ž 3a2n q b . 3

2

580

SOLUTIONS TO SELECTED EXERCISES

Thus a2nq1 - b, and the sequence is bounded from above by 'b . ii. The sequence is monotone increasing: we have that Ž3bq a2n .rŽ3a2n q b . ) 1, since a2n - b, as was seen earlier. Hence, a nq1 ) a n for all n. By Corollary 5.1.1Ž1., the sequence must be convergent. Let c be its limit. We then have the equation cs

2.

c Ž 3bq c 2 . 3c 2 q b

,

which results from taking the limit of both sides of a nq1 s a nŽ3bq a2n .rŽ3a2n q b . and noting that lim n™⬁ a nq1 s lim n™⬁ a n s c. The only solution to the above equation is c s 'b . b- 1. In this case, we can similarly show that the sequence is bounded from below and is monotone decreasing. Therefore, by Corollary 5.1.1Ž2. it must be convergent. Its limit is equal to 'b .

5.8. The sequence is bounded from above by 3, that is, a n - 3 for all n: This is true for n s 1. If it is true for n, then a nq1 s Ž 2 q a n .

1r2

- Ž 2 q 3.

1r2

- 3.

By induction, a n - 3 for all n. Furthermore, the sequence is monotone increasing: a1 F a2 , since a1 s 1, a2 s '3 . If a n F a nq1 , then a nq2 s Ž 2 q a nq1 .

1r2

G Ž 2 q an .

1r2

s a nq1 .

By induction, a n F a nq1 for all n. By Corollary 5.1.1Ž1. the sequence must be convergent. Let c be its limit, which can be obtained by solving the equation c s Ž2 qc.

1r2

.

The only solution is c s 2. 5.10. Let m s 2 n. Then a m y a n s 1rŽ n q 1. q 1rŽ n q 2. q ⭈⭈⭈ q1r2 n. Hence, am y an )

n 2n

s

1 2

.

Therefore, a m y a n cannot be made less than 12 no matter how large m and n are, if m s 2 n. This violates the condition for a sequence to be Cauchy.

581

CHAPTER 5

5.11. For m ) n, we have that a m y a n s Ž a m y a my1 . q Ž a my1 y a my2 . q ⭈⭈⭈ q Ž a nq2 y a nq1 . q Ž a nq1 y a n . - br my1 q br my2 q ⭈⭈⭈ qbr n s br n Ž 1 q r q r 2 q ⭈⭈⭈ qr myny1 . s

br n Ž 1 y r myn . 1yr

-

br n 1yr

.

For a given ⑀ ) 0, we choose n large enough such that br nrŽ1 y r . - ⑀ , which implies that a m y a n - ⑀ , that is, the sequence satisfies the Cauchy criterion; hence, it is convergent. 5.12. If
sn4⬁ns1 is bounded from above, then it must be convergent, since it is monotone increasing and thus Ý⬁ns1 a n converges. Vice versa, if Ý⬁ns1 a n is convergent, then
sn4⬁ns1 is a convergent sequence; hence, it is bounded, by Theorem 5.1.1. 5.13. Let sn be the nth partial sum of the series. Then 1

sn s

3

n

Ý

is1

ž ž

1 1

s

ž

1 3i y 1

y

3 2 1 1

s

q

5 y

3 2 ™

1

1 6

y 1

1 3i q 2

y

5

1 8

1 3n q 2

/

q ⭈⭈⭈ q

1 3n y 1

y

1 3n q 2

/

/

as n ™ ⬁.

5.14. For n ) 2, the binomial theorem gives

ž

1q

1 n

/

n

s1q s2q

ž/ ž/

1 n 1 n 1 q q ⭈⭈⭈ q n 2 1 n 2 n n

1 2! n

-2q

Ý

is2

ž

1y 1 i!

n

-2q

Ý

is2

-3 - n.

1 2

iy1

1 n

/

q

1 3!

ž

1y

1 n

/ž

1y

2 n

/

q ⭈⭈⭈ q

1 nn

582

SOLUTIONS TO SELECTED EXERCISES

Hence,

ž

1q

1 n

/

- n1r n , 1

p

Ž n1r n y 1 . )

.

np

Therefore, the series Ý⬁ns1 Ž n1r n y 1. p is divergent by the comparison test, since Ý⬁ns1 1rn p is divergent for pF 1. 5.15. (a) Suppose that a n - M for all n, where M is a positive constant. Then an

)

1 q an

an 1qM

.

The series Ý⬁ns1 a nrŽ1 q a n . is divergent by the comparison test, since Ý⬁ns1 a n is divergent. (b) We have that an

1

s1y

1 q an

1 q an

.

If
a4⬁ns1 is not bounded, then lim a n s ⬁,

hence,

n™⬁

an

lim

1 q an

n™⬁

s 1 / 0.

Therefore, Ý⬁ns1 a nrŽ1 q a n . is divergent. 5.16. The sequence
sn4⬁ns1 is monotone increasing. Hence, for n s 2, 3, . . . , an sn2 n

Ý

ai

2 is1 s i

F s

an

s

sn sny1 a1 s12

F

a1

s

a1

s12 s12

n

q

Ý

is2

q

q

ž

1 s1

1 s1

sn y sny1 sn sny1

s

1 sny1

y

1 sn

,

ai si2 y

y

1 s2

1 sn

/

q ⭈⭈⭈ q

™

a1 s12

q

ž

1 s1

1 sny1

y

1 sn

/

,

since sn ™ ⬁ by the divergence of Ý⬁ns1 a n . It follows that Ý⬁ns1 a nrsn2 is a convergent series.

583

CHAPTER 5

5.17. Let A denote the sum of the series Ý⬁ns1 a n . Then rn s A y sny1 , where sn s Ý nis1 a i . The sequence
rn4⬁ns2 is monotone decreasing. Hence, n

Ý

ai

ism

)

ri

a m q a mq1 q ⭈⭈⭈ qa ny1 rm

s1y

rn rm

s

rm y rn rm

.

Since rn ™ 0, we have 1 y rnrrm ™ 1 as n ™ ⬁. Therefore, for 0 - ⑀ - 1, there exists a positive integer k such that am rm

q

a mq1 rmq1

q ⭈⭈⭈ q

a mqk rmqk

)⑀ .

This implies that the series Ý⬁ns1 a nrrn does not satisfy the Cauchy criterion. Hence, it is divergent. 5.18. Ž1rn.logŽ1rn. s yŽlog n.rn™ 0 as n ™ ⬁. Hence, Ž1rn.1r n ™ 1. Similarly, Ž1rn.logŽ1rn2 . s yŽ2 log n.rn™ 0, which implies Ž1rn2 .1r n ™ 1. n 5.19. (a) a1r s n1r n y 1 ™ 0 - 1 as n ™ ⬁. Therefore, Ý⬁ns1 a n is convergent n by the root test. 2 (b) a n - wlogŽ1 q n.xrlog e n s wlogŽ1 q n.xrn2 . Since

⬁

H1

log Ž 1 q x . x

2

⬁

1

dxsy log Ž 1 q x . q x 1 slog 2 q log

ž

x xq 1

/

⬁

H1

dx x Ž xq 1 .

⬁ 1

s 2 log 2, the series Ý⬁ns1 wlogŽ1 q n.xrn2 is convergent by the integral test. Hence, Ý⬁ns1 a n converges by the comparison test. (c) a nra nq1 s Ž2 n q 2.Ž2 n q 3.rŽ2 n q 1. 2 ´ lim n™⬁ n Ž a nra nq1 y 1. s 32 ) 1. The series Ý⬁ns1 a n is convergent by Raabe’s test. (d)

'

a n s n q 2'n y 'n s

n q 2'n y n

'n q 2'n

q 'n

Therefore, Ý⬁ns1 a n is divergent.

™1

as n ™ ⬁.

584

SOLUTIONS TO SELECTED EXERCISES

(e) < a n < 1r n s 4rn™ 0 - 1. The series is absolutely covergent by the root test. (f) a n s Žy1. n sinŽ␲rn.. For large n, sinŽ␲rn. ; ␲rn. Therefore, Ý⬁ns1 a n is conditionally convergent. 5.20. (a) Applying the ratio test, we have the condition a nq1

< x < 2 lim

an

n™⬁

- 1,

which is equivalent to < x< 2 3

- 1,

< x < - '3 .

that is,

The series is divergent if < x < s '3 . Hence, it is uniformly convergent on wyr, r x where r - '3 . (b) Using the ratio test, we get x lim

a nq1 an

n™⬁

- 1,

where a n s 10 nrn. This is equivalent to 10 x - 1. The series is divergent if < x < s 101 . Hence, it is uniformly convergent on wyr, r x where r - 101 . (c) The series is uniformly convergent on wyr, r x where r - 1. (d) The series is uniformly convergent everywhere by Weierstrass’s M-test, since cos nx n Ž n q 1. 2

F

1 n Ž n q 1. 2

for all x,

and the series whose nth term is 1rnŽ n2 q 1. is convergent. 5.21. The series Ý⬁ns1 a n is convergent by Theorem 5.2.14. Let s denote its sum: s s 1 y 12 q 13 y Ž 14 y 15 . y Ž 16 y 17 . y ⭈⭈⭈ - 1 y 12 q 13 s 56 s 10 12 .

585

CHAPTER 5

Now, ⬁

Ý bn s Ž 1 q 13 y 12 . q Ž 15 q 17 y 14 . q ⭈⭈⭈

ns1

q

ž

1 4ny3

q

1 4ny1

y

1 2n

/

q ⭈⭈⭈ .

Let sX3 n denote the sum of the first 3n terms of Ý⬁ns1 bn . Then, sX3 n s s2 n q u n , where s2 n is the sum of the first 2 n terms of Ý⬁ns1 a n , and un s

1 2nq1

q

1 2nq3

q ⭈⭈⭈ q

1 4ny1

,

n s 1, 2, . . .

The sequence
u n4⬁ns1 is monotone increasing and is bounded from above by 12 , since un -

n 2nq1

,

n s 1, 2, . . .

Žthe number of terms that make up u n is equal to n.. Thus
sX3 n4⬁ns1 is convergent, which implies convergence of Ý⬁ns1 bn . Let t denote the sum of this series. Note that t ) Ž 1 q 13 y 12 . q Ž 15 q 17 y 14 . s 11 12 , since 1rŽ4 n y 3. q 1rŽ4 n y 1. y 1r2 n ) 0 for all n. 5.22. Let c n denote the nth term of Cauchy’s product. Then n

cn s

Ý ak anyk

ks0

s Ž y1 .

n

n

Ý

ks0

1

Ž n y k q 1. Ž k q 1.

1r2

.

586

SOLUTIONS TO SELECTED EXERCISES

Since n y k q 1 F n q 1 and k q 1 F n q 1, n

1

Ý

cn G

nq1

ks0

s 1.

Hence, c n does not go to zero as n ™ ⬁. Therefore, Ý⬁ns0 c n is divergent. 5.23. f nŽ x . ™ f Ž x ., where f Ž x. s

½

xs 0, x) 0.

0, 1rx,

The convergence is not uniform on w0, ⬁., since f nŽ x . is continuous on w0, ⬁. for all n, but f Ž x . is discontinuous at xs 0. 5.24. (a) 1rn x F 1rn1q ␦ . Since Ý⬁ns1 1rn1q ␦ is a convergent series, Ý⬁ns1 1rn x is uniformly convergent on w1 q ␦ , ⬁. by Weierstrass’s M-test. (b) drdx Ž1rn x . s yŽlog n.rn x. The series Ý⬁ns1 Žlog n.rn x is uniformly convergent on w1 q ␦ , ⬁.: log n n

x

F

log n

1 1q␦ r2

n ␦ r2

n

,

xG ␦ q 1.

Since Žlog n.rn ␦ r2 ™ 0 as n ™ ⬁, there exists a positive integer N such that log n n ␦ r2

-1

if n ) N.

Therefore, log n n

x

-

1 1q␦ r2

n

if n ) N and xG ␦ q 1. But, Ý⬁ns1 1rn1q␦ r2 is convergent. Hence, Ý⬁ns1 Žlog n.rn x is uniformly convergent on w1 q ␦ , ⬁. by Weierstrass’s M-test. If ␨ Ž x . denotes the sum of the series Ý⬁ns1 1rn x , then

␨ X Ž x. sy

⬁

log n

ns1

nx

Ý

,

x G ␦ q 1.

587

CHAPTER 5

5.25. We have that

ž

⬁

Ý

ns0

/

1 nqky1 n x s , k n Ž1yx.

k s 1,2, . . . ,

if y1 - x- 1 Žsee Example 5.4.4.. Differentiating this series term by term, we get ⬁

Ý nx ny1

ns1

ž

/

k nqky1 s . kq1 n Ž1yx.

It follows that ⬁

Ýn

ns0

ž

/

r Ž1 y p. n nqry1 r p Ž1 y p. s pr r p rq1 s

r Ž1 y p. p

.

Taking now second derivatives, we obtain

ž

⬁

Ý n Ž n y 1. x ny2

ns2

/

k Ž k q 1. nqky1 s . kq2 n Ž1yx.

From this we conclude that ⬁

Ý n2

ns1

ž

/

n q k y 1 n kx Ž 1 q kx . x s . kq2 n Ž1yx.

Hence, ⬁

Ý n2

ns0

ž

/

pr Ž1 y p. r 1 q r Ž1 y p. n nqry1 r p Ž1 y p. s n p rq2 s

5.26.

␾Ž t. s

⬁

Ý e nt

ns0

spr

⬁

Ý

ns0

s

ž

pr 1 y qe t

ž

r Ž 1 y p . Ž 1 q r y rp . p2

/

nqry1 r n pq n

nqry1 n ,

/Ž

qe t .

qe t - 1.

n

.

588

SOLUTIONS TO SELECTED EXERCISES

The series converges uniformly on Žy⬁, s x where s - ylog q. Yes, formula Ž5.63. can be applied, since there exists a neighborhood N␦ Ž0. contained inside the interval of convergence by the fact that ylog q ) 0. 5.28. It is sufficient to show that the series Ý⬁ns1 Ž ␮Xnrn!.␶ n is absolutely convergent for some ␶ ) 0 Žsee Theorem 5.6.1.. Using the root test,

␳ s lim sup n™⬁

␮Xn

1rn

Ž n! .

s ␶ e lim sup n™⬁

s ␶ e lim sup

␶

1rn

␮Xn

Ž 2␲ . ␮Xn

1r2 n

n1q1r2 n

1rn

.

n

n™⬁

1rn

Let m s lim sup n™⬁

␮Xn

1rn

.

n

If m ) 0 and finite, then the series Ý⬁ns1 Ž ␮Xnrn!.␶ n is absolutely convergent if ␳ - 1, that is, if ␶ - 1rŽ em.. 5.30. (a) E Ž Xn . s

ž/ Ý ž /

n

n k nyk p Ž1 y p. k

Ýk

ks0

n

sp

k

ks1

ny1

sp

Ý Ž k q 1.

ks0

ny1

sp

n ky1 nyk p Ž1 y p. k

Ý

ks0

ž

/

n nyky1 pk Ž1 y p. kq1

n! k! Ž n y k y 1 . !

s np Ž pq 1 y p .

pk Ž1 y p.

ny1

s np. We can similarly show that E Ž X n2 . s np Ž 1 y p . q n2 p 2 .

nyky1

589

CHAPTER 5

Hence, Var Ž X n . s E Ž X n2 . y n2 p 2 Var Ž X n . s np Ž 1 y p . . (b) We have that P Ž Yn G r␴ . F 1rr 2 . Let r␴ s ⑀ , where ␴ 2 s pŽ1 y p .rn. Then P Ž Yn G ⑀ . F

pŽ1 y p. n⑀ 2

.

(c) As n ™ ⬁, P Ž Yn G ⑀ . ™ 0, which implies that Yn converges in probability to zero. 5.31. (a)

␾n Ž t . s Ž pn e t q qn . , n

qn s 1 y pn

s 1 q pn Ž e t y 1 .

n

.

Let npn s rn . Then

␾n Ž t . s 1 q

rn n

Ž e t y 1.

n

™ exp ␮ Ž e t y 1 . as n ™ ⬁. The limit of ␾nŽ t . is the moment generating function of a Poisson distribution with mean ␮. Thus S n has a limiting Poisson distribution with mean ␮. 5.32. We have that Q s Ž I n y H 1 q kH 2 y k 2 H 3 q k 3 H 4 y ⭈⭈⭈ .

2

s Ž I n y H 1 . q k 2 H 3 y 2 k 3 H 4 q ⭈⭈⭈ . Thus y X Qys y X Ž I n y H 1 . y q k 2 y X H 3 y y 2 k 3 y X H 4 y q ⭈⭈⭈ s SS E q k 2 S3 y 2 k 3 S4 q ⭈⭈⭈ s SS E q

⬁

Ý Ž i y 2. Ž yk . iy1 Si .

is3

590

SOLUTIONS TO SELECTED EXERCISES

CHAPTER 6 6.1. If f Ž x . is Riemann integrable, then inequality Ž6.1. is satisfied; hence, equality Ž6.6. is true, as was shown to be the case in the proof of Theorem 6.2.1. Vice versa, if equality Ž6.6. is satisfied, then by the double inequality Ž6.2., S Ž P, f . must have a limit as ⌬ p ™ 0, which implies that f Ž x . is Riemann integrable on w a, b x. 6.2. Consider the function f Ž x . on w0, 1x such that f Ž x . s 0 if 0 F x - 12 , f Ž x . s 2 if x s 1, and f Ž x . s Ý nis0 1r2 i if Ž n q 1.rŽ n q 2. F x Ž n q 2.rŽ n q 3. for n s 0, 1, 2, . . . . This function has a countable number of discontinuities at 12 , 23 , 34 , . . . , but is Riemann integrable on w0, 1x, since it is monotone increasing Žsee Theorem 6.3.2.. 6.3. Suppose that f Ž x . has one discontinuity of the first kind at x s c, a- c - b. Let lim x ™ cy f Ž x . s L1 , lim x ™ cq f Ž x . s L2 , L1 / L2 . In any partition P of w a, b x, the point c appears in at most two subintervals. The contribution to US P Ž f . y LSP Ž f . from these intervals is less than 2Ž My m. ⌬ p , where M, m are the supremum and infimum of f Ž x . on w a, b x, and this can be made as small as we please. Furthermore, Ý i Mi ⌬ x i y Ý i m i ⌬ x i for the remaining subintervals can also be made as small as we please. Hence, US P Ž f . y LSP Ž f . can be made smaller than ⑀ for any given ⑀ ) 0. Therefore, f Ž x . is Riemann integrable on w a, b x. A similar argument can be used if f Ž x . has a finite number of discontinuities of the first kind in w a, b x. 6.4. Consider the following partition of w0, 1x: Ps
0, 1r2 n, 1rŽ2 n y 1., . . . , 1r3, 1r2, 14 . The number of partition points is 2 n q 1 including 0 and 1. In this case, 2n

Ý

⌬ fi s

is1

1 2n q

q

q

cos ␲ n y 0 1 2ny1 1 2ny2 1 2

s

1 n

q

ny1

2

Ž 2 n y 1. y

cos ␲ Ž n y 1 . y

cos ␲ y 1

␲

cos

1 3

q

cos

ž /

1 ny2

3␲ 2

1 2n

cos ␲ n

1 2ny1

q cos

q ⭈⭈⭈ q1.

cos

ž / ␲ 2

y

␲ 2 1 2

Ž 2 n y 1 . q ⭈⭈⭈

cos ␲

591

CHAPTER 6

n As n ™ ⬁, Ý2is1 ⌬ f i ™ ⬁, since Ý⬁ns1 1rn is divergent. Hence, f Ž x . is not of bounded variation on w0, 1x.

6.5. (a) For a given ⑀ ) 0, there exists a constant M) 0 such that fXŽ x. gX Ž x .

yL -⑀

if x) M. Since g X Ž x . ) 0, f X Ž x . y Lg X Ž x . - ⑀ g X Ž x . . Hence, if ␭1 and ␭2 are chosen larger than M, then

H␭

␭2

f X Ž x . y Lg X Ž x . dx F

1

H␭

␭2 1

H␭

-⑀

f X Ž x . y Lg X Ž x . dx

␭2 X

g Ž x . dx.

1

(b) From Ža. we get f Ž ␭2 . y Lg Ž ␭2 . y f Ž ␭1 . q Lg Ž ␭1 . - ⑀ g Ž ␭2 . y g Ž ␭1 . . Divide both sides by g Ž ␭2 . wwhich is positive for large ␭2 , since g Ž x . ™ ⬁ as x™ ⬁x, we obtain f Ž ␭2 . g Ž ␭2 .

yLy

f Ž ␭1 . g Ž ␭2 .

qL

g Ž ␭1 . g Ž ␭2 .

-⑀ 1y

g Ž ␭1 . g Ž ␭2 .

-⑀ . Hence, f Ž ␭2 . g Ž ␭2 .

yL -⑀q

f Ž ␭1 . g Ž ␭2 .

q L

g Ž ␭1 . g Ž ␭2 .

.

(c) For sufficiently large ␭ 2 , the second and third terms on the right-hand side of the above inequality can each be made smaller than ⑀ ; hence, f Ž ␭2 . g Ž ␭2 .

y L - 3⑀ .

6.6. We have that m g Ž x. Ff Ž x. g Ž x. FM g Ž x. ,

592

SOLUTIONS TO SELECTED EXERCISES

where m and M are the infimum and supremum of f Ž x . on w a, b x, respectively. Let ␰ and ␩ be points in w a, b x such that m s f Ž ␰ ., Ms f Ž␩ .. We conclude that

Ha f Ž x . g Ž x . dx b

fŽ ␰ . F

Ha g Ž x . dx b

Ff Ž␩ . .

By the intermediate-value theorem ŽTheorem 3.4.4., there exists a constant c, between ␰ and ␩ , such that

Ha f Ž x . g Ž x . dx b

Ha

b

g Ž x . dx

sf Ž c. .

Note that c can be equal to ␰ or ␩ in case equality is attained at the lower end or the upper end of the above double inequality. 6.9. Integration by parts gives

Ha f Ž x . dg Ž x . s f Ž b . g Ž b . y f Ž a. g Ž a. q Ha g Ž x . d yf Ž x . b

b

.

Since g Ž x . is bounded, f Ž b . g Ž b . ™ 0 as b™ ⬁. Let us now establish convergence of Hab g Ž x . dwyf Ž x .x as b™ ⬁: let M) 0 be such that < g Ž x .< F M for all xG a. In addition,

Ha M d yf Ž x . b

s Mf Ž a . y Mf Ž b . ™ Mf Ž a .

as b™ ⬁.

Hence, Ha⬁M dwyf Ž x .x is a convergent integral. Since yf Ž x . is monotone increasing on w a, ⬁., then ⬁

Ha

g Ž x . d yf Ž x . F

⬁

Ha M d yf Ž x .

.

This implies absolute convergence of Ha⬁ g Ž x . dwyf Ž x .x. It follows that the integral Ha⬁ f Ž x . dg Ž x . is convergent.

593

CHAPTER 6

6.10. Let n be a positive integer. Then n␲

H0

sin x x

dxs

␲

H0

sin x

dxy

x

q ⭈⭈⭈ q Ž y1 . 1

␲

s

H0 sin x

)

ž

1

␲

2

s

␲

q

ž

x 1 1 2

2␲

ny1

q

sin x

dx

x n␲

HŽ ny1 .␲

1 xq␲

q ⭈⭈⭈ q

2␲

1q

H␲

q ⭈⭈⭈ q

/H

/

dx 1

xq Ž n y 1 . ␲

dx

␲

n␲ n

x

q ⭈⭈⭈ q

1 1

sin x

sin x dx

0

™⬁

as n ™ ⬁.

6.11. (a) This is convergent by the integral test, since H1⬁ Žlog x .rŽ x'x . dxs 4H0⬁eyx x dxs 4 by a proper change of variable. (b) Ž n q 4.rŽ2 n3 q 1. ; 1r2 n2 . But Ý⬁ns1 1rn2 is convergent by the integral test, since H1⬁ dxrx 2 s 1. Hence, this series is convergent by the comparison test. (c) 1rŽ'n q 1 y 1. ; 1r 'n . By the integral test, Ý⬁ns1 1r 'n is diver⬁ gent, since H1⬁ dxr'x s 2'x 1 s ⬁. Hence, Ý⬁ns1 1rŽ'n q 1 y 1. is divergent. 6.13. (a) Near xs 0 we have x my1 Ž1 y x . ny1 ; x my1 , and H01 x my1 dxs 1rm. Also, near x s 1 we have x my1 Ž1 y x . ny1 ; Ž1 y x . ny1 , and H01 Ž1 y x . ny1 dxs 1rn. Hence, B Ž m, n. converges if m ) 0, n ) 0. (b) Let 'x s sin ␪ . Then ␲

H0 x

1 my1

Ž1yx.

ny1

H0 2 sin

dxs 2

␪ cos 2 ny1␪ d␪ .

2 my1

(c) Let xs 1rŽ1 q y .. Then

H0 x

1 my1

Ž1yx.

ny1

dxs

y ny1

⬁

H0

Ž1qy.

mqn

dy.

Letting zs 1 y x, we get

H0 x

1 my1

Ž1yx.

ny1

H1 Ž 1 y z .

dxs y s

H0 x

0

1 ny1

my1

Ž1yx.

z ny1 dz

my1

dx.

594

SOLUTIONS TO SELECTED EXERCISES

Hence, B Ž m, n. s B Ž n, m.. It follows that B Ž m, n . s

x my1

⬁

H0

Ž1qx.

mqn

dx.

(d) x ny1

⬁

B Ž m, n . s

H0

s

H0

mqn

dx

mqn

dxq

Ž1qx. x ny1

1

Ž1qx.

x ny1

⬁

H1

Ž1qx.

mqn

dx.

Let y s 1rx in the second integral. We get x ny1

⬁

H1

Ž1qx.

dxs

mqn

H0

y my1

1

Ž1qy.

mqn

dy

Therefore,

6.14. (a)

B Ž m, n . s

H0

s

H0

⬁

x ny1

1

Ž1qx.

1

x ny1 q x my1

Ž1qx.

dx

H0 '1 q x

dxq

mqn

s 3

1

Ž1qx.

mqn

dx

dx.

mqn

dx

H0 '1 q x 1

x my1

H0

q 3

⬁

dx

H1 '1 q x

3

.

The first integral exists because the integrand is continuous. The second integral is convergent because dx

⬁

H1 '1 q x

3

-

⬁

H1

dx x 3r2

s 2.

Hence, H0⬁dxr '1 q x 3 is convergent. (b) Divergent, since for large x we have 1rŽ1 q x 3 .1r3 ; 1rŽ1 q x ., and H0⬁ dxrŽ1 q x . s ⬁. (c) Convergent, since if f Ž x . s 1rŽ1 y x 3 .1r3 and g Ž x . s 1rŽ1 y x .1r3 , then limy

x™1

f Ž x. gŽ x.

s

ž / 1

3

1r3

.

But H01 g Ž x . dxs 32 . Hence, H01 f Ž x . dx is convergent.

595

CHAPTER 6

(d) Partition the integral as the sum of

H0

1

dx

'x Ž 1 q x .

dx

⬁

H1

and

'x Ž 1 q 2 x .

.

The first integral is convergent, since near x s 0 we have 1rw'x Ž1 q x .x ; 1r 'x , and H01 dxr 'x s 2. The second integral is also convergent, since as x ™ ⬁ we have 1rw'x Ž1 q 2 x .x ; 1r2 x 3r2 , and H1⬁ dxrx 3r2 s 2. 6.15. The proof of this result is similar to the proof of Corollary 6.4.2. 6.16. It is is easy to show that

Ž by x . f Ž n. Ž x . . Ž x. sy Ž n y 1. ! ny1

hXn Hence,

h n Ž a. s h n Ž b . y s

1

Ha h Ž x . dx b X

n

H Ž by x . Ž n y 1. ! a b

ny1

f Ž n. Ž x . dx,

since h nŽ b . s 0. It follows that

Ž by a. f Ž ny1. Ž a . Ž n y 1. ! ny1

f Ž b . s f Ž a . q Ž by a . f X Ž a . q ⭈⭈⭈ q q

1

H Ž by x . Ž n y 1. ! a b

ny1

f Ž n. Ž x . dx.

6.17. Let G Ž x . s Hax g Ž t . dt. Then, G Ž x . is uniformly continuous and GX Ž x . s g Ž x . by Theorem 6.4.8. Thus

Ha f Ž x . g Ž x . dxsf Ž x . G Ž x . b

b ay

s f Ž b. GŽ b. y

Ha G Ž x . f Ž x . dx b

X

Ha G Ž x . f Ž x . dx. b

X

Now, since G Ž x . is continuous, then by Corollary 3.4.1, GŽ ␰ . FGŽ x. FGŽ␩ .

596

SOLUTIONS TO SELECTED EXERCISES

for all x in w a, b x, where ␰ , ␩ are points in w a, b x at which G Ž x . achieves its infimum and supremum in w a, b x, respectively. Furthermore, since f Ž x . is monotone Žsay monotone increasing . and f X Ž x . exists, then f X Ž x . G 0 by Theorem 4.2.3. It follows that GŽ ␰ .

Ha f Ž x . dxF Ha G Ž x . f Ž x . dxF G Ž ␩ . Ha f Ž x . dx. b X

b

b X

X

This implies that

Ha G Ž x . f Ž x . dxs ␭Ha f Ž x . dx, b

b X

X

where G Ž ␰ . F ␭ F G Ž␩ .. By Theorem 3.4.4, there exists a point c between ␰ and ␩ such that ␭ s GŽ c .. Hence,

Ha G Ž x . f Ž x . dxs G Ž c . Ha f Ž x . dx b

b X

X

Ha g Ž x . dx. c

s f Ž b . y f Ž a. Consequently,

Ha f Ž x . g Ž x . dxs f Ž b . G Ž b . y Ha G Ž x . f Ž x . dx b

b

X

s f Ž b.

Ha g Ž x . dxy

s f Ž a.

Ha g Ž x . dxq f Ž b . Hc g Ž x . dx

b

f Ž b . y f Ž a.

c

Ha g Ž x . dx c

b

6.18. This follows directly from applying the result in Exercise 6.17 and letting f Ž x . s 1rx, g Ž x . s sin x.

Ha

b

sin x x

dxs s

1

1

H sin x dxq b Hc sin x dx a a

1 a

c

b

1

Ž cos ay cos c . q Ž cos c y cos b . . b

Therefore,

Ha

b

sin x x

dx F F

1

cos ay cos c q

a 4 a

.

1 a

cos c y cos b

597

CHAPTER 6

6.21. Let ⑀ ) 0 be given. Choose ␩ ) 0 such that

␩ g Ž b . y g Ž a. - ⑀ . Since f Ž x . is uniformly continuous on w a, b x Žsee Theorem 3.4.6., there exists a ␦ ) 0 such that f Ž x. yf Ž z. -␩ if xy z - ␦ for all x, z in w a, b x. Let us now choose a partition P of w a, b x whose norm ⌬ p is smaller than ␦ . Then, US P Ž f , g . y LSP Ž f , g . s

n

Ý Ž Mi y m i . ⌬ g i

is1

n

F ␩ Ý ⌬ gi is1

s ␩ g Ž b . y g Ž a. -⑀ . It follows that f Ž x . is Riemann᎐Stieltjes integrable on w a, b x. 6.22. Since g Ž x . is continuous, for any positive integer n we can choose a partition P such that ⌬ gi s

g Ž b . y g Ž a. n

i s 1, 2, . . . , n.

,

Also, since f Ž x . is monotone increasing, then Mi s f Ž x i ., m i s f Ž x iy1 ., i s 1, 2, . . . , n. Hence, US P Ž f , g . y LSP Ž f , g . s s

g Ž b . y g Ž a. n g Ž b . y g Ž a. n

n

Ý

f Ž x i . y f Ž x iy1 .

is1

f Ž b . y f Ž a. .

This can be made less than ⑀ , for any given ⑀ ) 0, if n is chosen large enough. It follows that f Ž x . is Riemann᎐Stieltjes integrable with respect to g Ž x .. 6.23. Let f Ž x . s x krŽ1 q x 2 ., g Ž x . s x ky2 . Then lim x ™⬁ f Ž x .rg Ž x . s 1. The ⬁ integral H0⬁ x ky2 dx is divergent, since H0⬁ x ky2 dxsx ky1 r Ž k y 1 . 0 s ⬁ if k ) 1. If k s 1, then H0⬁ x ky2 dxs H0⬁ dxrxs ⬁. By Theorem 6.5.3, H0⬁ f Ž x . dx must be divergent if k G 1. A similar procedure can be used 0 to show that Hy⬁ f Ž x . dx is divergent.

598

SOLUTIONS TO SELECTED EXERCISES

6.24. Since E Ž X . s 0, then VarŽ X . s E Ž X 2 ., which is given by EŽ X 2 . s

x2e x

⬁

Hy⬁ Ž 1 q e

dx.

.

x 2

Let u s e xrŽ1 q e x .. The integral becomes EŽ X 2 . s

H0

1

log

½ ž

2

du

u log u q Ž 1 y u . log Ž 1 y u .

u log u q Ž 1 y u . log Ž 1 y u .

1

uŽ 1 y u. log u

H0

1

1yu

H0

s y2

/

1yu

H0

sy

/

1yu u

s log y

ž

u

1

duy

log u 1yu

H0

1

log Ž 1 y u . u

5

du

du

du.

But

H0

1

log u

dus

1yu

H0 Ž 1 q u q u q ⭈⭈⭈ qu 1

2

n

q ⭈⭈⭈ . log u du,

and

H0 u

1 n

log u dus

1

1 nq1

sy

u nq1 log u y 0

1

Ž n q 1.

2

1

Hu nq1 0

1 n

.

Hence,

H0

1

log u 1yu

dus y

⬁

1

ns0

Ž n q 1.

Ý

2

sy

and E Ž X 2 . s ␲ 2r3. 6.26. Let g Ž x . s f Ž x .rx ␣ , hŽ x . s 1rx ␣. We have that limq

x™0

gŽ x. hŽ x .

s f Ž 0 . - ⬁.

␲2 6

,

du

1 0

599

CHAPTER 6

Since H0␦ hŽ x . dxs ␦ 1y␣ rŽ1 y ␣ . - ⬁ for any ␦ ) 0, by Theorem 6.5.7, H0␦ f Ž x .rx ␣ dx exists. Furthermore, ⬁

H␦

f Ž x. x

1

⬁

␣

H␦

E Ž Xy␣ . s

H0

dxF

␣

␦

f Ž x . dxF

1

␦␣

.

Hence, ⬁

f Ž x. x␣

dx

exists. 6.27. Let g Ž x . s f Ž x .rx 1q␣ , hŽ x . s 1rx. Then limq

x™0

gŽ x. hŽ x .

s k - ⬁.

Since H0␦ hŽ x . dx is divergent for any ␦ ) 0, then so is H0␦ g Ž x . dx by Theorem 6.5.7, and hence E w XyŽ1q␣ . x s

⬁

H0 g Ž x . dxq H␦ ␦

g Ž x . dxs ⬁.

6.28. Applying formula Ž6.84., the density function of W is given by

g Ž w. s

2⌫

ž

nq1

'n␲ ⌫

/

ž /ž

2

n

1q

2

w2 n

/

y Ž nq1 .r2

w G 0.

,

6.29. (a) From Theorem 6.9.2, we have P Ž X y ␮ G u␴ . F

1 u2

.

Letting u ␴ s r, we get PŽ Xy␮ Gr . F

␴2 r2

.

(b) This follows from Ža. and the fact that E Ž Xn . s ␮ ,

Var Ž X n . s

␴2 n

.

600

SOLUTIONS TO SELECTED EXERCISES

(c ) P Ž Xn y ␮ G ⑀ . F

␴2 n⑀ 2

.

Hence, as n ™ ⬁, P Ž X n y ␮ G ⑀ . ™ 0. 6.30. (a) Using the hint, for any u and ®) 0 we have u 2␯ ky1 q 2 u®␯ k q ® 2␯ kq1 G 0. Since ␯ ky1 G 0 for k G 1, we must therefore have ® 2␯ k2 y ® 2␯ ky1 ␯ kq1 F 0,

k G 1,

that is,

␯ k2 F ␯ ky1 ␯ kq1 ,

k s 1, 2, . . . , n y 1.

(b) ␯ 0 s 1. For k s 1, ␯ 12 F ␯ 2 . Hence, ␯ 1 F ␯ 21r2 . Let us now use 1rŽ nq1. mathematical induction to show that ␯n1r n F ␯nq1 for all n for which ␯n and ␯nq1 exist. The statement is true for n s 1. Suppose that 1rŽ ny1. ␯ny1 F ␯n1r n . 1rŽ nq1. To show that ␯n1r n F ␯nq1 : We have that

␯n2 F ␯ny1 ␯nq1 F ␯nŽ ny1. r n␯nq1 . Thus,

␯nŽ nq1. r n F ␯nq1 . Hence, 1rŽ nq1. ␯n1r n F ␯nq1 .

CHAPTER 7 7.1. (a) Along x 1 s 0, we have f Ž0, x 2 . s 0 for all x 2 , and the limit is zero as x ™ 0. Any line through the origin can be represented by the equation x 1 s t x 2 . Along this line, we have f Ž t x2 , x2 . s s

t x2 x 22 t x2

ž

exp y

ž

exp y

t

x2 x 22

t x2

/

,

/

x 2 / 0.

Using l’Hospital’s rule ŽTheorem 4.2.6., f Ž t x 2 , x 2 . ™ 0 as x 2 ™ 0.

601

CHAPTER 7

(b) Along the parabola x 1 s x 22 , we have

ž /

x 22

f Ž x1 , x 2 . s

exp y

x 22

s ey1 ,

x 22

x 22

x 2 / 0,

which does not go to zero as x 2 ™ 0. By Definition 7.2.1, f Ž x 1 , x 2 . does not have a limit as x ™ 0. 7.5. (a) This function has no limit as x ™ 0, as was shown in Example 7.2.2. Hence, it is not continuous at the origin. (b)

⭸ f Ž x1 , x 2 . ⭸ x1

⌬ x 1™0

xs0

⭸ f Ž x1 , x 2 . ⭸ x2

½ ½

s lim

s lim

⌬ x 2™0

xs0

1

0⌬ x 1

⌬ x1

2 Ž ⌬ x1 . q 0

1

0⌬ x 2

⌬ x2 0 q Ž ⌬ x2 . 2

y0

y0

5 5

s 0,

s 0.

7.6. Applying formula Ž7.12., we get df dt

k

s

Ý xi

⭸ f Ž u.

s nt ny1 f Ž x . ,

⭸ ui

is1

where u i is the ith element of u s t x, i s 1, 2, . . . , k. But, on one hand,

⭸ f Ž u. ⭸ xi

k

Ý

s

⭸ f Ž u. ⭸ u j ⭸ uj

js1

st

⭸ f Ž u. ⭸ ui

⭸ xi

,

and on the other hand,

⭸ f Ž u. ⭸ xi

stn

⭸ f Ž x. ⭸ xi

.

Hence,

⭸ f Ž u. ⭸ ui

s t ny1

⭸ f Ž x. ⭸ xi

.

It follows that k

Ý xi

is1

⭸ f Ž u. ⭸ ui

s nt ny1 f Ž x .

602

SOLUTIONS TO SELECTED EXERCISES

can be written as k

Ý xi

t ny1

⭸ f Ž x. ⭸ xi

is1

s nt ny1 f Ž x . ,

that is, k

Ý xi

⭸ f Ž x. ⭸ xi

is1

s nf Ž x . .

7.7. (a) f Ž x 1 , x 2 . is not continuous at the origin, since along x 1 s 0 and x 2 / 0, f Ž x 1 , x 2 . s 0, which has a limit equal to zero as x 2 ™ 0. But, along the parabola x 2 s x 12 , f Ž x1 , x 2 . s

x 14 x 14 q x 14

1

s

if x 1 / 0.

2

Hence, lim x 1 ™ 0 f Ž x 1 , x 12 . s 12 / 0. (b) The directional derivative in the direction of the unit vector v s Ž ®1 , ®2 .X at the origin is given by ®1

⭸ f Ž x.

q®2

⭸ x1

⭸ f Ž x.

xs0

⭸ x2

. xs0

This derivative is equal to zero, since

⭸ f Ž x. ⭸ x1

s lim xs0

⭸ f Ž x. ⭸ x2

⌬ x 1™0

s lim xs0

1

2 Ž ⌬ x1 . 0

⌬ x1

4 Ž ⌬ x1 . q 0

1

0⌬ x 2

⌬ x2 0 q Ž ⌬ x2 . 2

⌬ x 2™0

y 0 s 0,

y 0 s 0.

7.8. The directional derivative of f at a point x on C in the direction of v is Ý kis1 ®i ⭸ f Žx.r⭸ x i . But v is given by vs

s

s

dx

dx

dt

dt

dx dt

½

dg i Ž t .

k

Ý

dt

is1

dx

dg

dt

dt

,

2

5

1r2

603

CHAPTER 7

where g is the vector whose ith element is g i Ž t .. Hence, k

Ý ®i

⭸ f Ž x. ⭸ xi

is1

dg i Ž t . ⭸ f Ž x .

k

1

s

Ý

dgrdt

⭸ xi

dt

is1

.

Furthermore, ds

dg

s

dt

.

dt

It follows that k

Ý ®i

⭸ f Ž x. ⭸ xi

is1

Ý

ds k

dx i ⭸ f Ž x .

7.9. (a)

ž

f Ž x 1 , x 2 . f f Ž 0, 0 . q x 1

df Ž x .

q

1 2!

ž

x 12

.

ds

⭸ ⭸ x1

⭸2

⭸ xi

ds

is1

s

q x2

⭸ ⭸ x2

q 2 x1 x 2

⭸ x 12

⭸ xi

dt

is1

Ý

s

dg i Ž t . ⭸ f Ž x .

k

dt

s

/

f Ž 0, 0 .

⭸2 ⭸ x1 ⭸ x 2

q x 22

⭸2 ⭸ x 22

s 1 q x1 x 2 . (b) f Ž x 1 , x 2 , x 3 . f f Ž 0, 0, 0 . q q

1 2!

ž

⭸

3

Ý xi ⭸ x

is1

žÝ / ⭸

3

xi

is1

⭸ xi

ž

f Ž x 1 , x 2 . f f Ž 0, 0 . q x 1 q

1 2!

s 1,

ž

x 12

⭸ x1

⭸ x 12

f Ž 0, 0, 0 .

f Ž 0, 0, 0 .

⭸

⭸2

/

f Ž 0, 0 .

2

s sin Ž 1 . q x 1 cos Ž 1 . q (c )

i

/

1 2!


x 12

q x2

q 2 x1 x 2

cos Ž 1 . y sin Ž 1 . q 2 x 22 cos Ž 1 . 4 .

⭸ ⭸ x2

/

f Ž 0, 0 .

⭸2 ⭸ x1 ⭸ x 2

q x 22

⭸2 ⭸ x 22

/

f Ž 0, 0 .

since all first-order and second-order partial derivatives vanish at x s 0.

604

SOLUTIONS TO SELECTED EXERCISES

7.10. (a) If u1 s f Ž x 1 , x 2 . and ⭸ fr⭸ x 1 / 0 at x 0 , then by the implicit function theorem ŽTheorem 7.6.2., there is neighborhood of x 0 in which the equation u1 s f Ž x 1 , x 2 . can be solved uniquely for x 1 in terms of x 2 and u1 , that is, x 1 s hŽ u1 , x 2 .. Thus, f h Ž u1 , x 2 . , x 2 ' u1 . Hence, by differentiating this identity with respect to x 2 , we get

⭸ f ⭸h ⭸ x1 ⭸ x 2

q

⭸f ⭸ x2

s 0,

which gives

⭸h ⭸ x2

sy

⭸f

⭸f

⭸ x2

⭸ x1

in a neighborhood of x 0 . (b) On the basis of part Ža., we can consider g Ž x 1 , x 2 . as a function of x 2 and u, since x 1 s hŽ u1 , x 2 . in a neighborhood of x 0 . In such a neighborhood, the partial derivative of g with respect to x 2 is

⭸g ⭸h ⭸ x1 ⭸ x 2

q

⭸g ⭸ x2

s

⭸Ž f, g.

⭸f

⭸ Ž x1 , x 2 .

⭸ x1

,

which is equal to zero because ⭸ Ž f, g .r⭸ Ž x 1 , x 2 . s 0. wRecall that ⭸ hr⭸ x 2 s yŽ ⭸ fr⭸ x 2 .Ž ⭸ hr⭸ x 1 .x. (c) From part Žb., g w hŽ u1 , x 2 ., x 2 x is independent of x 2 in a neighborhood of x 0 . We can then write ␾ Ž u1 . s g w hŽ u1 , x 2 ., x 2 x. Since x 1 s hŽ u1 , x 2 . is equivalent to u1 s f Ž x 1 , x 2 . in a neighborhood of x 0 , ␾ w f Ž x 1 , x 2 .x s g Ž x 1 , x 2 . in this neighborhood. (d) Let G Ž f, g . s g Ž x 1 , x 2 . y ␾ w f Ž x 1 , x 2 .x. Then, in a neighborhood of x 0 , GŽ f, g . s 0. Hence,

⭸G ⭸ f ⭸ f ⭸ x1 ⭸G ⭸ f ⭸ f ⭸ x2

q

q

⭸G ⭸g ⭸ g ⭸ x1 ⭸G ⭸g ⭸ g ⭸ x2

' 0, ' 0.

In order for these two identities to be satisfied by values of ⭸ Gr⭸ f, ⭸ Gr⭸ g not all zero, it is necessary that ⭸ Ž f, g .r⭸ Ž x 1 , x 2 . be equal to zero in this neighborhood of x 0 .

605

CHAPTER 7

7.11. uŽ x 1 , x 2 , x 3 . s uŽ ␰ 1 ␰ 3 , ␰ 2 ␰ 3 , ␰ 3 .. Hence,

⭸u ⭸␰ 3

s

⭸ u ⭸ x1 ⭸ x 1 ⭸␰ 3

s␰1

⭸u ⭸ x1

q

q␰2

⭸ u ⭸ x2 ⭸ x 2 ⭸␰ 3 ⭸u ⭸ x2

q

q

⭸u ⭸ x3

⭸ u ⭸ x3 ⭸ x 3 ⭸␰ 3

,

so that

␰3

⭸u ⭸␰ 3

s␰1 ␰3 s x1

⭸u ⭸ x1

⭸u ⭸ x1

q␰2 ␰3

q x2

⭸u ⭸ x2

⭸u ⭸ x2

q x3

q␰3

⭸u ⭸ x3

⭸u ⭸ x3

s nu.

Integrating this partial differential equation with respect to ␰ 3 , we get log u s nlog ␰ 3 q ␺ Ž ␰ 1 , ␰ 2 . , where ␺ Ž ␰ 1 , ␰ 2 . is a function of ␰ 1 , ␰ 2 . Hence, u s ␰ 3n exp ␺ Ž ␰ 1 , ␰ 2 . s x 3n F Ž ␰ 1 , ␰ 2 . s x 3n F

ž

/

x1 x 2 , , x3 x3

where F Ž ␰ 1 , ␰ 2 . s expw ␺ Ž ␰ 1 , ␰ 2 .x. 7.13. (a) The Jacobian determinant is 27x 12 x 22 x 32 , which is zero in any subset of R 3 that contains points on any of the coordinate planes. (b) The unique inverse function is given by x 2 s u1r3 2 ,

x 1 s u1r3 1 ,

x 3 s u1r3 3 .

7.14. Since ⭸ Ž g 1 , g 2 .r⭸ Ž x 1 , x 2 . / 0, we can solve for x 1 , x 2 uniquely in terms of y 1 and y 2 by Theorem 7.6.2. Differentiating g 1 and g 2 with respect to y 1 , we obtain

⭸ g1 ⭸ x1 ⭸ x1 ⭸ y1 ⭸ g 2 ⭸ x1 ⭸ x1 ⭸ y1

q q

⭸ g1 ⭸ x2 ⭸ x 2 ⭸ y1 ⭸ g2 ⭸ x2 ⭸ x 2 ⭸ y1

q q

⭸ g1 ⭸ y1 ⭸ g2 ⭸ y1

s 0, s 0,

Solving for ⭸ x 1r⭸ y 1 and ⭸ x 2r⭸ y 1 yields the desired result.

606

SOLUTIONS TO SELECTED EXERCISES

7.15. This is similar to Exercise 7.14. Since ⭸ Ž f, g .r⭸ Ž x 1 , x 2 . / 0, we can solve for x 1 , x 2 in terms of x 3 , and we get dx 1 dx 3 dx 2 dx 3

sy

sy

⭸Ž f, g.

⭸Ž f, g.

⭸ Ž x3 , x2 .

⭸ Ž x1 , x 2 .

⭸Ž f, g.

⭸Ž f, g.

⭸ Ž x1 , x 3 .

⭸ Ž x1 , x 2 .

,

.

But

⭸Ž f, g. ⭸ Ž x3 , x2 .

sy

⭸Ž f, g. ⭸ Ž x2 , x3 .

⭸Ž f, g.

,

⭸ Ž x1 , x 3 .

sy

⭸Ž f, g. ⭸ Ž x 3 , x1 .

.

Hence, dx 1

⭸ Ž f , g . ⭸ Ž x2 , x3 .

s

dx 3

⭸ Ž f , g . ⭸ Ž x1 , x 2 .

s

dx 2

⭸ Ž f , g . ⭸ Ž x 3 , x1 .

.

7.16. (a) Žy 13 , y 13 . is a point of local minimum. (b)

⭸f ⭸ x1 ⭸f ⭸ x2

s 4␣ x 1 y x 2 q 1 s 0, s yx 1 q 2 x 2 y 1 s 0.

The solution of these two equations is x 1 s 1rŽ1 y 8 ␣ ., x 2 s Ž4␣ y 1.rŽ8 ␣ y 1.. The Hessian matrix of f is As

4␣ y1

y1 . 2

Here, f 11 s 4␣ , and detŽA. s 8 ␣ y 1. i. Yes, if ␣ ) 18 . ii. No, it is not possible. iii. Yes, if ␣ - 18 , ␣ / 0. (c) The stationary points are Žy2, y2., Ž4, 4.. The first is a saddle point, and the second is a point of local minimum. (d) The stationary points are Ž'2 , y '2 ., Žy '2 , '2 ., and Ž0, 0.. The first two are points of local minimum. At Ž0, 0., f 11 f 22 y f 122 s 0. In

607

CHAPTER 7

this case, hXAh has the same sign as f 11 h s Ž h1 , h 2 .X , except when h1 q h 2

f 12 f 11

Ž0, 0. s y4

for all values of

s0

or h1 y h 2 s 0, in which case hXAh s 0. For such values of h, ŽhX ⵱ . 3 f Ž0, 0. s 0, but ŽhX ⵱ . 4 f Ž0, 0. s 24Ž h14 q h 42 . s 48 h14 , which is nonnegative. Hence, the point Ž0, 0. is a saddlepoint. 7.19. The Lagrange equations are

Ž 2 q 8 ␭ . x 1 q 12 x 2 s 0, 12 x 1 q Ž 4 q 2 ␭ . x 2 s 0, 4 x 12 q x 22 s 25. We must have

Ž 4␭ q 1 . Ž ␭ q 2 . y 36 s 0. The solutions are ␭1 s y4.25, ␭ 2 s 2. For ␭1 s y4.25, we have the points i. x 1 s 1.5, x 2 s 4.0, ii. x 1 s y1.5, x 2 s y4.0. For ␭2 s 2, we have the points iii. x 1 s 2, x 2 s y3, iv. x 1 s y2, x 2 s 3. The matrix B 1 has the value 2q8␭ 12 B1 s 8 x1

12 4q2␭ 2 x2

8 x1 2 x2 . 0

The determinant of B 1 is ⌬ 1. At Ži., ⌬ 1 s 5000; the point is a local maximum. At Žii., ⌬ 1 s 5000; the point is a local maximum. At Žiii. and Živ., ⌬ 1 s y5000; the points are local minima.

608

SOLUTIONS TO SELECTED EXERCISES

7.21. Let F s x 12 x 22 x 32 q ␭Ž x 12 q x 22 q x 32 y c 2 .. Then

⭸F ⭸ x1 ⭸F ⭸ x2 ⭸F ⭸ x3

s 2 x 1 x 22 x 32 q 2 ␭ x 1 s 0, s 2 x 12 x 2 x 32 q 2 ␭ x 2 s 0, s 2 x 12 x 22 x 3 q 2 ␭ x 3 s 0,

x 12 q x 22 q x 32 s c 2 . Since x 1 , x 2 , x 3 cannot be equal to zero, we must have x 22 x 32 q ␭ s 0, x 12 x 32 q ␭ s 0, x 12 x 22 q ␭ s 0. These equations imply that x 12 s x 22 s x 32 s c 2r3 and ␭ s yc 4r9. For these values, it can be verified that ⌬ 1 - 0, ⌬ 2 ) 0, where ⌬ 1 and ⌬ 2 are the determinants of B 1 and B 2 , and

B1 s

B2 s

2 x 22 x 32 q 2 ␭

4 x 1 x 2 x 32

4 x 1 x 22 x 3

2 x1

4 x 1 x 2 x 32

2 x 12 x 32 q 2 ␭

4 x 12 x 2 x 3

2 x2

x 12 x 22 q 2 ␭

2 x3 0

4 x 1 x 22

4 x 12 x 2

x3

x3

2

2 x1

2 x2

2 x 12 x 32 q 2 ␭

4 x 12 x 2 x 3

2 x2

x 12 x 22 q 2 ␭

2 x3 . 0

4 x 12 x 2

x3

2

2 x2

2 x3

2 x3

,

Since n s 3 is odd, the function x 12 x 22 x 32 must attain a maximum value given by Ž c 2r3. 3. It follows that for all values of x 1 , x 2 , x 3 , x 12 x 22

x 32 F

ž / c2

3

,

3

that is,

Ž x 12 x 22 x 32 .

1r3

F

x 12 q x 22 q x 32 3

.

609

CHAPTER 7

7.24. The domain of integration, D, is the region bounded from above by the parabola x 2 s 4 x 1 y x 12 and from below by the parabola x 2 s x 12 . The two parabolas intersect at Ž0, 0. and Ž2, 4.. It is then easy to see that

H0 Hx4 x yx f Ž x , x 2

1

1

H2y'x'4yx

. dx 2 dx1 s H

4

2 1

2 1

2

f Ž x 1 , x 2 . dx 1 dx 2 .

2

0

2

'1yx 2 f Ž x 1 , x 2 . dx1 x dx 2 . 7.25. (a) I s H01 w H1yx 2

2

1yx 1 (b) dgrdx1 s H1yx 1

⭸ f Ž x1 , x 2 . ⭸ x1

dx 2 y 2 x 1 f Ž x 1 , 1 y x 12 . q f Ž x 1 , 1 y x 1 ..

7.26. Let u s x 22rx 1 , ®s x 12rx 2 . Then u®s x 1 x 2 , and

HHDx

1 x2

dx 1 dx 2 s

⭸ Ž x1 , x 2 .

H1 H1 u® 2

2

du d®.

⭸ Ž u, ® .

But

⭸ Ž x1 , x 2 . ⭸ Ž u, ® .

y1

⭸ Ž u, ® .

s

s

⭸ Ž x1 , x 2 .

1 3

.

Hence,

HHDx

dx 1 dx 2 s

1 x2

3 4

.

7.28. Let I Ž a. s H0'3 dxrŽ aq x 2 .. Then d 2 I Ž a. da

2

H0'3

s2

dx

Ž aq x 2 .

3

.

On the other hand, I Ž a. s d 2 I Ž a. da

2

s

1

'a 3 4

Arctan

(

3 a

ay5r2 Arctan

q

(

3 a

'3 Ž 2 aq 3. 2 a2 Ž aq 3 . 2

.

q

'3

1

4 a Ž aq 3 . 2

610

SOLUTIONS TO SELECTED EXERCISES

Thus

H0'3

dx 2 3

Ž aq x .

s

3 8

ay5r2 Arctan

q

(

3 a

'3 Ž 2 aq 3. 4 a2 Ž aq 3 . 2

q

'3

1

8 a Ž aq 3 . 2

.

Putting as 1, we obtain

H0'3

dx

Ž1qx2 .

3

s

3 8

Arctan'3 q

7'3 64

.

7.29. (a) The marginal density functions of X 1 and X 2 are f 1Ž x1 . s

H0 Ž x q x 1

1

2

s x 1 q 12 , f2 Ž x2 . s

. dx 2 0 - x 1 - 1,

H0 Ž x q x 1

1

s x 2 q 12 ,

2

. dx1 0 - x 2 - 1,

f Ž x1 , x 2 . / f 1Ž x1 . f 2 Ž x 2 . . The random variables X 1 and X 2 are therefore not independent. (b) E Ž X1 X 2 . s

H0 H0 x 1

1

1 x2

Ž x 1 q x 2 . dx1 dx 2

s 13 .

7.30. The marginal densities of X 1 and X 2 are

°1 q x , ¢0,

f Ž x . s~1 y x , 1

1

1

f2 Ž x2 . s 2 x2 ,

1

y1 - x 1 - 0, 0 - x 1 - 1, otherwise, 0 - x 2 - 1.

611

CHAPTER 7

Note that f Ž x 1 , x 2 . / f 1Ž x 1 . f 2 Ž x 2 .. Hence, X 1 and X 2 are not independent. But E Ž X1 X 2 . s

H0 Hyx x x2

1

1 x2

dx 1 dx 2 s 0,

2

E Ž X1 . s

Hy1 x Ž 1 q x . dx q H0 x Ž 1 y x . dx s 0,

E Ž X2 . s

H0 2 x

0

1

1

1

2 2

1

1

1

1

1

dx 2 s 23 .

Hence, E Ž X 1 X 2 . s E Ž X 1 . E Ž X 2 . s 0. 7.31. (a) The joint density function of Y1 and Y2 is g Ž y1 , y 2 . s

1 ⌫Ž ␣ . ⌫Ž ␤ .

y 1␣y1 Ž 1 y y 1 .

␤y1

y 2␣q␤y1 eyy 2 .

(b) The marginal densities of Y1 and Y2 are, respectively, g 1Ž y1 . s g2 Ž y2 . s

⌫Ž ␣q␤ . ⌫Ž ␣ . ⌫Ž ␤ . BŽ ␣ , ␤ . ⌫Ž ␣ . ⌫Ž ␤ .

y 1␣y1 Ž 1 y y 1 .

␤y1

y 2␣q␤y1 eyy 2 ,

,

0 - y 1 - 1,

0 - y 2 - ⬁.

(c) Since B Ž ␣ , ␤ . s ⌫ Ž ␣ . ⌫ Ž ␤ .r⌫ Ž ␣ q ␤ ., g Ž y1 , y 2 . s g 1Ž y1 . g 2 Ž y 2 . , and Y1 and Y2 are therefore independent. 7.32. Let U s X 1 , W s X 1 X 2 . Then, X 1 s U, X 2 s WrU. The joint density function of U and W is g Ž u, w . s

10 w 2 u2

,

w - u - 'w ,

0 - w - 1.

612

SOLUTIONS TO SELECTED EXERCISES

The marginal density function of W is g 1 Ž w . s 10 w 2 s 10 w 2

du

Hw'w u

ž

1

2

y

w

/

1

'w

0 - w - 1.

,

7.34. The inverse of this transformation is X 1 s Y1 , X 2 s Y2 y Y1 , . . . X n s Yn y Yny1 , and the absolute value of the Jacobian determinant of this transformation is 1. Therefore, the joint density function of the Yi ’s is g Ž y 1 , y 2 , . . . , yn . s eyy 1 eyŽ y 2yy 1 . ⭈⭈⭈ eyŽ y nyy ny1 . s eyy n ,

0 - y 1 - y 2 - ⭈⭈⭈ - yn - ⬁.

The marginal density function of Yn is g n Ž yn . s eyy n s

H0 H0 yn

ynny1

Ž n y 1. !

yny1

⭈⭈⭈

H0

eyy n ,

y2

dy1 dy 2 ⭈⭈⭈ dyny1

0 - yn - ⬁.

7.37. Let ␤ s Ž ␤ 0 , ␤ 1 , ␤ 2 .X . The least-squares estimate of ␤ is given by

ˆ s Ž XX X . ␤

y1

XX y,

where y s Ž y 1 , y 2 , . . . , yn .X , and X is a matrix of order n = 3 whose first column is the column of ones, and whose second and third columns are given by the values of x i , x i2 , i s 1, 2, . . . , n. 7.38. Maximizing LŽx, p. is equivalent to maximizing its natural logarithm. Let us therefore consider maximizing log L subject to Ý kis1 pi s 1. Using the method of Lagrange multipliers, let F s log L q ␭

ž

k

Ý pi y 1

is1

/

.

613

CHAPTER 8

Differentiating F with respect to p1 , p 2 , . . . , pk , and equating the derivatives to zero, we obtain, x irpi q ␭ s 0 for i s 1, 2, . . . , k. Combining these equations with the constraint Ý kis1 pi s 1, we obtain the maximum likelihood estimates

ˆpi s

xi n

i s 1 ,2, . . . , k.

,

CHAPTER 8 8.1. Let x i s Ž x i1 , x i2 .X , h i s Ž h i1 , h i2 .X , i s 0, 1, 2, . . . . Then f Ž x i q t h i . s 8 Ž x i1 q th i1 . y 4 Ž x i1 q th i1 . Ž x i2 q th i2 . q 5 Ž x i2 q th i2 . 2

2

s a i t 2 q bi t q c i , where a i s 8 h2i1 y 4 h i1 h i2 q 5h2i2 , bi s 16 x i1 h i1 y 4 Ž x i1 h i2 q x i2 h i1 . q 10 x i2 h i2 , 2 c i s 8 x i12 y 4 x i1 x i2 q 5 x i2 .

The value of t that minimizes f Žx i q t h i . in the direction of h i is given by the solution of ⭸ fr⭸ t s 0, namely, t i s ybirŽ2 a i .. Hence, x iq1 s x i y

bi 2 ai

hi,

i s 0, 1, 2, . . . ,

,

i s 0, 1, 2, . . . ,

where hisy

⵱f Ž x i . ⵱f Ž x i .

2

and X

⵱f Ž x i . s Ž 16 x i1 y 4 x i2 , y4 x i1 q 10 x i2 . ,

i s 0, 1, 2, . . . .

614

SOLUTIONS TO SELECTED EXERCISES

The results of the iterative minimization procedure are given in the following table: Iteration i

xi

hi

ti

ai

f Žx i .

0 1 2 3 4 5 6

Ž5, 2.X Ž0.5, 2.X Ž0.5, 0.2.X Ž0.05, 0.2.X Ž0.05, 0.02.X Ž0.005, 0.02.X Ž0.005, 0.002.X

Žy1, 0.X Ž0, y1.X Žy1, 0.X Ž0, y1.X Žy1, 0.X Ž0, y1.X Žy1, 0.X

4.5 1.8 0.45 0.18 0.045 0.018 0.0045

8 5 8 5 8 5 8

180 18 1.8 0.18 0.018 0.0018 0.00018

Note that a i ) 0, which confirms that t i s ybirŽ2 a i . does indeed minimize f Žx i q t h i . in the direction of h i . It is obvious that if we were to continue with this iterative procedure, the point x i would converge to 0. 8.3. (a)

ˆy Ž x . s 45.690 q 4.919 x 1 q 8.847x 2 y 0.270 x 1 x 2 y 4.148 x 12 y 4.298 x 22 .

ˆ is (b) The matrix B ˆs B

y4.148 y0.135

y0.135 . y4.298

ˆ a Its eigenvalues are ␶ 1 s y8.136, ␶ 2 s y8.754. This makes B negative definite matrix. The results of applying the method of ridge analysis inside the region R are given in the following table: ␭

x1

x2

r

ˆy

31.4126 13.5236 7.5549 4.5694 2.7797 1.5873 0.7359 0.0967 y0.4009 y0.7982

0.0687 0.1373 0.2059 0.2745 0.3430 0.4114 0.4797 0.5480 0.6163 0.6844

0.1236 0.2473 0.3709 0.4946 0.6184 0.7421 0.8659 0.9898 1.1137 1.2376

0.1414 0.2829 0.4242 0.5657 0.7072 0.8485 0.9899 1.1314 1.2729 1.4142

47.0340 48.2030 49.1969 50.0158 50.6595 51.1282 51.4218 51.5404 51.4839 51.2523

Note that the stationary point, x 0 s Ž0.5601, 1.0117.X , is a point of ˆ are negative. This absolute maximum since the eigenvalues of B point falls inside R. The corresponding maximum value of ˆ y is 51.5431.

615

CHAPTER 8

8.4. We know that

Ž Bˆ y ␭1 I k . x 1 s y Ž Bˆ y ␭2 I k . x 2 s y

1 2

ˆ, ␤

1 2

ˆ, ␤

where x 1 and x 2 correspond to ␭1 and ␭2 , respectively, and are such that x X1 x 1 s x X2 x 2 s r 2 , with r 2 being the common value of r 12 and r 22 . The corresponding values of ˆ y are

ˆ q xX1 Bx ˆ 1, ˆy 1 s ␤ˆ0 q xX1 ␤ ˆ q xX2 Bx ˆ 2. ˆy 2 s ␤ˆ0 q xX2 ␤ We then have

ˆ 1 y xX2 Bx ˆ 2 q 12 Ž xX1 y xX2 . ␤ ˆ s Ž ␭1 y ␭2 . r 2 , x X1 Bx ˆ q Ž ␭1 y ␭2 . r 2 . ˆy 1 y ˆy 2 s 12 Ž xX1 y xX2 . ␤ Furthermore, from the equations defining x 1 and x 2 , we have

ˆ Ž ␭2 y ␭1 . xX1 x 2 s 12 Ž xX1 y xX2 . ␤. Hence,

ˆy 1 y ˆy 2 s Ž ␭1 y ␭2 . Ž r 2 y xX1 x 2 . . But r 2 s 5x 1 5 2 5x 2 5 2 ) x X1 x 2 , since x 1 and x 2 are not parallel vectors. We conclude that ˆ y1 )ˆ y 2 whenever ␭1 ) ␭ 2 . 8.5.

If MŽx 1 . is positive definite, then ␭1 must be smaller than all the ˆ Žthis is based on applying the spectral decomposition eigenvalues of B ˆ theorem to B and using Theorem 2.3.12.. Similarly, if MŽx 2 . is indefiˆ and also nite, then ␭2 must be larger than the smallest eigenvalue of B smaller than its largest eigenvalue. It follows that ␭1 - ␭ 2 . Hence, by Exercise 8.4, ˆ y1 -ˆ y2 .

8.6. (a) We know that

Ž Bˆ y ␭I k . x s y

1 2

ˆ ␤.

Differentiating with respect to ␭ gives

⭸x

Ž Bˆ y ␭I k . ⭸␭ s x,

616

SOLUTIONS TO SELECTED EXERCISES

and since x X x s r 2 ,

⭸x

xX

sr

⭸␭

⭸r ⭸␭

.

A second differentiation with respect to ␭ yields

⭸ 2x

Ž Bˆ y ␭I k . xX

⭸ 2x

q

⭸␭2

⭸␭

⭸x

s2

2

⭸ xX ⭸ x

⭸␭ ⭸ 2r

sr

⭸␭ ⭸␭

,

ž / ⭸r

q

⭸␭2

2

⭸␭

.

If we premultiply the second equation by ⭸ 2 x Xr⭸␭2 and the fourth equation by ⭸ x Xr⭸␭, subtract, and transpose, we obtain xX

⭸ 2x

y2

⭸␭2

⭸ xX ⭸ x

s 0.

⭸␭ ⭸␭

Substituting this in the fifth equation, we get r

⭸ 2r ⭸␭2

s3

⭸ xX ⭸ x

y

⭸␭ ⭸␭

ž / ⭸r

2

⭸␭

.

Now, since

⭸r ⭸␭

s

⭸ ⭸␭

s xX

Ž xX x .

1r2

⭸ xr⭸␭

Ž xX x .

1r2

,

we conclude that r

3

⭸ 2r ⭸␭2

s 3r

2

⭸ xX ⭸ x ⭸␭ ⭸␭

ž /

y xX

⭸x ⭸␭

2

.

(b) The expression in Ža. can be written as r3

⭸ 2r ⭸␭2

s2r2

⭸ xX ⭸ x ⭸␭ ⭸␭

q r2

⭸ xX ⭸ x ⭸␭ ⭸␭

ž /

y xX

⭸x ⭸␭

2

.

The first part on the right-hand side is nonnegative and is zero only when r s 0 or when ⭸ xr⭸␭ s 0. The second part is nonnegative by

617

CHAPTER 8

the fact that

ž / xX

⭸x ⭸␭

2

Fr2

⭸x ⭸␭

2 2

X

sr2

⭸x ⭸x ⭸␭ ⭸␭

.

Equality occurs only when x s 0, that is, r s 0, or when ⭸ xr⭸␭ s 0. But when ⭸ xr⭸␭ s 0, we have x s 0 if ␭ is different from all the ˆ and thus r s 0. It follows that ⭸ 2 rr⭸␭2 ) 0 except eigenvalues of B, when r s 0, where it takes the value zero. 8.7. (a) Bs s

n⍀

␴2 n ␴2

HR
E

ˆy Ž x . y ␩ Ž x . 4 dx 2


Ž ␥ y ␤ .X ⌫11 Ž ␥ y ␤ . y 2 Ž ␥ y ␤ .X ⌫12 ␦ q ␦X ⌫22 ␦ 4 ,

where ⌫11 , ⌫12 , ⌫22 are the region moments defined in Section 8.4.3. (b) To minimize B we differentiate it with respect to ␥ , equate the derivative to zero, and solve for ␥. We get 2 ⌫11 Ž ␥ y ␤ . y 2 ⌫12 ␦ s 0, y1 ␥ s ␤ q ⌫11 ⌫12 ␦

s C␶ . This solution minimizes B, since ⌫11 is positive definite. (c) B is minimized if and only if ␥ s C␶. This is equivalent to stating that C␶ is estimable, since E Žˆ␭ . s ␥. (d) Writing ˆ␭ as a linear function of the vector y of observations of the form ˆ␭ s Ly, we obtain E Ž ˆ␭ . s L E Ž y . s L Ž X␤ q Z␦ . s L wX : Zx ␶ . But ␥ s E Žˆ␭ . s C␶. We conclude that C s LwX : Zx. (e) It is obvious from part Žd. that the rows of C are spanned by the rows of wX : Zx.

618

SOLUTIONS TO SELECTED EXERCISES

(f) The matrix L defined by L s ŽXX X.y1 XX satisfies the equation L w X : Z x s C, since L w X : Z x s Ž XX X .

y1

XX w X : Z x

s I : Ž XX X .

y1

XX Z

s I : My1 11 M 12 y1 s I : ⌫11 ⌫12

s C. 8.8. If the region R is a sphere of radius 1, then 3 g 2 F 1. Now,

⌫11 s

1 0

0 1 5

0 0

0 0

0

0

1 5

0

0

0

0

1 5

0 0 ⌫12 s 0 0

0 0 0 0

,

0 0 . 0 0

Hence, C s wI 4 : O4=3 x. Furthermore, X s w1 4 : Dx ,

Zs

g2

g2

g2

g2

yg 2

yg 2

yg 2

g2

yg 2

yg 2

yg 2

g2

.

Hence, M 11 s 14 XX X

s

1 0

0 g2

0 0

0 0

0

0

g2

0

0

0

0

g2

,

619

CHAPTER 8

M 12 s 14 XX Z

s

(a )

0 0

0 0

0 yg 3

0

yg 3

0

yg 3

0

0

M 11 s ⌫11

´

g 2 s 15

M 12 s ⌫12

´

gs0

.

Thus, it is not possible to choose g so that D satisfies the conditions in Ž8.56.. (b) Suppose that there exists a matrix L of order 4 = 4 such that C s L wX : Zx . Then I 4 s LX, 0 s LZ. The second equation implies that L is of rank 1, while the first equation implies that the rank of L is greater than or equal to the rank of I 4 , which is equal to 4. Therefore, it is not possible to find a matrix such as L. Hence, g cannot be chosen so that D satisfies the minimum bias property described in part Že. of Exercise 8.7. 8.9. (a) Since ⌬ is symmetric, it can be written as ⌬ s P⌳ P X , where ⌳ is a diagonal matrix of eigenvalues of ⌬ and the columns of P are the corresponding orthogonal eigenvectors of ⌬ , each of length equal to 1 Žsee Theorem 2.3.10.. It is easy to see that over the region ␺ , h Ž ␦ , D . F ␦X ␦ emax Ž ⌬ . F r 2 emax Ž ⌬ . , by the fact that emax Ž ⌬ .I y P⌳ P X is positive semidefinite. Without loss of generality, we consider that the diagonal elements of ⌳ are written in descending order. The upper bound in the above inequality is attained by hŽ ␦, D. for ␦ s r P1 , where P1 is the first column of P, which corresponds to emax Ž ⌬ .. (b) The design D can be chosen so that emax Ž ⌬ . is minimized over the region R. 8.10. This is similar to Exercise 8.9. Write T as P⌳ P X , where ⌳ is a diagonal matrix of eigenvalues of T, and P is an orthogonal matrix of corresponding eigenvectors. Then ␦X T ␦ s uX u, where u s ⌳1r2 P X ␦, and

620

SOLUTIONS TO SELECTED EXERCISES

␭Ž ␦, D. s ␦X S␦ s uX⌳y1r2 P X SP⌳y1r2 u. Hence, over the region ⌽, ␦X S␦ G ␬ emin Ž ⌳y1r2 P X SP⌳y1r2 . . But, if S is positive definite, then by Theorem 2.3.9, emin Ž ⌳y1r2 P X SP⌳y1r2 . s emin Ž P⌳y1 P X S. s emin Ž Ty1 S. . 8.11. (a) Using formula Ž8.52., it can be shown that Žsee Khuri and Cornell, 1996, page 229. Vs

ž

1

␭4 Ž k q 2.

␭4 Ž k q 2. y 2 k ␭2 y

y ␭22 k

q2 k

␭4 ␭2

/

␭4 Ž k q 1 . y ␭22 Ž k y 1 . ␭4 Ž k q 4.

,

where k is the number of input variables Žthat is, k s 2.,

␭2 s s s

␭4 s s s

n

1 n 1 n 8 n

Ý

i s 1, 2

us1

Ž2k q2 ␣ 2 . , n

1 n

2 x ui ,

Ý

2 2 x ui xu j ,

i/j

us1

2k n 4 n

,

and n s 2 k q 2 k q n 0 s 8 q n 0 is the total number of observations. Here, x ui denotes the design setting of variable x i , i s 1, 2; u s 1, 2, . . . , n. (b) The quantity V, being a function of n 0 , can be minimized with respect to n 0 .

621

CHAPTER 8

8.12. We have that X

X

ˆ q XB ˆ y XB . Ž Y y XB ˆ q XB ˆ y XB . Ž Y y XB . Ž Y y XB . s Ž Y y XB X

X

ˆ . Ž Y y XB ˆ . q Ž XB ˆ y XB . Ž XB ˆ y XB . , s Ž Y y XB since X

X

ˆ . Ž XB ˆ y XB . s Ž XX Y y XX XB ˆ. ŽB ˆ y B . s 0. Ž Y y XB ˆ y XB.X ŽXB ˆ y XB. is positive semidefinite, then Furthermore, since ŽXB by Theorem 2.3.19, X

X

ˆ . Ž Y y XB ˆ. , e i Ž Y y XB . Ž Y y XB . G e i Ž Y y XB

i s 1, 2, . . . , r ,

where e i Ž⭈. denotes the ith eigenvalue of a square matrix. If ŽY y ˆ .X ŽY y XB ˆ . are nonsingular, then by multiplyXB.X ŽY y XB. and ŽY y XB ing the eigenvalues on both sides of the inequality, we obtain X

X

ˆ . Ž Y y XB ˆ. . det Ž Y y XB . Ž Y y XB . G det Ž Y y XB ˆ Equality holds when B s B. 8.13. For any b, 1 q bF e b. Let as 1 q b. Then aF e ay1. Now, let ␭ i denote the ith eigenvalue of A. Then ␭ i G 0, and by the previous inequality,

ž

p

p

Ł ␭ i F exp Ý ␭i y p

is1

is1

/

.

Hence, det Ž A . F exp tr Ž A y I p . .

8.14. The likelihood function is proportional to det Ž V.

ynr2

exp y

n 2

tr Ž SVy1 . .

Now, by Exercise 8.13, det Ž SVy1 . F exp tr Ž SVy1 y I p . ,

622

SOLUTIONS TO SELECTED EXERCISES

since detŽSVy1 . s detŽVy1r2 SVy1r2 ., trŽSVy1 . s trŽVy1r2 SVy1r2 ., and Vy1r2 SVy1r2 is positive semidefinite. Hence, det Ž SVy1 .

nr2

exp y

n 2

tr Ž SVy1 . F exp y

n 2

tr Ž I p . .

This results in the following inequality: det Ž V.

ynr2

exp y

n 2

ynr2

tr Ž SVy1 . F det Ž S.

exp y

n 2

tr Ž I p . ,

which is the desired result.

ˆ where ␤ ˆ s ŽXX X.y1 XX y, and f X Žx. is as in model Ž8.47.. 8.16. ˆ y Žx. s f X Žx.␤, Simultaneous Ž1 y ␣ . = 100% confidence intervals on f X Žx.␤ for all x in R are of the form ˆ . Ž p MSE F␣ , p , nyp . f X Ž x. ␤

1r2

f X Ž x . Ž XX X.

y1

fŽ x.

1r2

.

For the points x 1 , x 2 , . . . , x m , the joint confidence coefficient is at least 1y␣.

CHAPTER 9 9.1. If f Ž x . has a continuous derivative on w0, 1x, then by Theorems 3.4.5 and 4.2.2 we can find a positive constant A such that f Ž x1 . y f Ž x 2 . F A x1 y x 2 for all x 1 , x 2 in w0, 1x. Thus, by Definition 9.1.2,

␻ Ž ␦ . F A␦ . Using now Theorem 9.1.3, we obtain f Ž x . y bn Ž x . F

3

A 1r2

2 n

for all x in w0, 1x. Hence, sup 0FxF1

where c s 32 A.

f Ž x . y bn Ž x . F

c 1r2

n

,

623

CHAPTER 9

9.2. We have that f Ž x1 . y f Ž x 2 . F

sup z1yz 2 F␦ 2

f Ž z1 . y f Ž z 2 .

for all x 1 y x 2 F ␦ 2 , and hence for all x 1 y x 2 F ␦ 1 , since ␦ 1 F ␦ 2 . It follows that f Ž x1 . y f Ž x 2 . F

sup x 1yx 2 F␦ 1

f Ž z1 . y f Ž z 2 . ,

sup z 1yz 2 F␦ 2

that is, ␻ Ž ␦ 1 . F ␻ Ž ␦ 2 .. 9.3. Suppose that f Ž x . is uniformly continuous on w a, b x. Then, for a given ⑀ ) 0, there exists a positive ␦ Ž ⑀ . such that f Ž x 1 . y f Ž x 2 . - ⑀ for all x 1 , x 2 in w a, b x for which x 1 y x 2 - ␦ . This implies that ␻ Ž ␦ . F ⑀ and hence ␻ Ž ␦ . ™ 0 as ␦ ™ 0. Vice versa, if ␻ Ž ␦ . ™ 0 as ␦ ™ 0, then for a given ⑀ ) 0, there exists ␦ 1 ) 0 such that ␻ Ž ␦ . - ⑀ if ␦ - ␦ 1. This implies that f Ž x1 . y f Ž x 2 . - ⑀

if x 1 y x 2 F ␦ - ␦ 1 ,

and f Ž x . must therefore be uniformly continuous on w a, b x. 9.4. By Theorem 9.1.1, there exists a sequence of polynomials, namely the Bernstein polynomials
bnŽ x .4⬁ns1 , that converges to f Ž x . s < x < uniformly on wya, ax. Let pnŽ x . s bnŽ x . y bnŽ0.. Then pnŽ0. s 0, and pnŽ x . converges uniformly to < x < on wya, ax, since bnŽ0. ™ 0 as n ™ ⬁. 9.5. The stated condition implies that H01 f Ž x . pnŽ x . dxs 0 for any polynomial pnŽ x . of degree n. In particular, if we choose pnŽ x . to be a Bernstein polynomial for f Ž x ., then it will converge uniformly to f Ž x . on w0, 1x. By Theorem 6.6.1,

H0

1

f Ž x.

2

dxs lim

n™⬁

H0 f Ž x . p Ž x . dx 1

n

s 0. Since f Ž x . is continuous, it must be zero everywhere on w0, 1x. If not, then by Theorem 3.4.3, there exists a neighborhood of a point in w0, 1x wat which f Ž x . / 0x on which f Ž x . / 0. This causes H01w f Ž x .x2 dx to be positive, a contradiction.

624

SOLUTIONS TO SELECTED EXERCISES

9.6. Using formula Ž9.15., we have f Ž x . y pŽ x . s

n

1

Ž n q 1. !

f Ž nq1. Ž c . Ł Ž xy a i . . is0

n But Ł is0 Ž xy ai . F n!Ž h nq1 r4. Žsee Prenter, 1975, page 37.. Hence,

f Ž x . y pŽ x . F

sup

␶nq1 h nq1

aFxFb

4 Ž n q 1.

.

9.7. Using formula Ž9.14. with a0 , a1 , a2 , and a3 , we obtain p Ž x . s l 0 Ž x . log a0 q l 1 Ž x . log a1 q l 2 Ž x . log a2 q l 3 Ž x . log a3 , where

l 0Ž x. s l 1Ž x . s l 2Ž x. s l 3Ž x . s

ž ž ž ž

xy a1 a0 y a1 xy a0 a1 y a0 xy a0 a2 y a0 xy a0 a3 y a0

/ž /ž /ž /ž

xy a2 a0 y a2 xy a2 a1 y a2 xy a1 a2 y a1 xy a1 a3 y a1

/ž /ž /ž /ž

xy a3 a0 y a3 xy a3 a1 y a3 x y a3 a2 y a3 x y a2 a3 y a2

/ / / /

,

,

,

.

Values of f Ž x . s log x and the corresponding values of pŽ x . at several points inside the interval w3.50, 3.80x are given below: x

f Ž x.

pŽ x .

3.50 3.52 3.56 3.60 3.62 3.66 3.70 3.72 3.77 3.80

1.25276297 1.25846099 1.26976054 1.28093385 1.28647403 1.29746315 1.30833282 1.31372367 1.327075 1.33500107

1.25276297 1.25846087 1.26976043 1.28093385 1.28647407 1.29746322 1.30833282 1.31372361 1.32707487 1.33500107

625

CHAPTER 9

Using the result of Exercise 9.6, an upper bound on the error of approximation is given by f Ž x . y pŽ x . F

sup 3.5FxF3 .8

␶4 h4 16

,

where h s max i Ž a iq1 y a i . s 0.10, and

␶4 s

f Ž4. Ž x .

sup 3.5FxF3 .8

y6

s

sup 3.5FxF3 .8

s

6

Ž 3.5 .

4

x4

.

Hence, the desired upper bound is

␶4 h4 16

ž /

6

s

0.10

16

4

3.5

s 2.5 = 10y7 . 9.8. We have that

Ha

b

f Y Ž x . y sY Ž x .

2

dxs

Ha

b

fYŽ x.

2

Ha s Ž x .

y2

b Y

dxy

Ha

b

sY Ž x .

2

dx

f Y Ž x . y sY Ž x . dx.

But integration by parts yields

Ha s Ž x . b Y

f Y Ž x . y sY Ž x . dx

ssY Ž x . f X Ž x . y sX Ž x .

b ay

Ha s

b Z

Ž x . f X Ž x . y xX Ž x . dx.

The first term on the right-hand side is zero, since f X Ž x . s sX Ž x . at xs a, b; and the second term is also zero, by the fact that sZ Ž x . is a constant, say sZ Ž x . s ␭ i , over Ž␶ i ,␶ iq1 .. Hence,

Ha s

b Z

Ž x . f X Ž x . y sX Ž x . dxs

ny1

␶ iq1

Ý ␭ iH

is0

␶i

f X Ž x . y sX Ž x . dxs 0.

626

SOLUTIONS TO SELECTED EXERCISES

It follows that

Ha

b

f Y Ž x . y sY Ž x .

2

dxs

Ha

b

fYŽ x.

2

dxy

Ha

b

sY Ž x .

2

dx,

which implies the desired result. 9.10.

⭸ rq1 h Ž x, ␪ . ⭸ x rq1

r

s Ž y1 . ␪ 1 ␪ 2 ␪ 3rq1 ey␪ 3 x .

Hence,

⭸ rq1 h Ž x, ␪ .

sup

⭸ x rq1

0FxF8

F 50.

Using inequality Ž9.36., the integer r is determined such that

ž /

2

8

rq1

Ž 50 . - 0.05.

Ž r q 1. ! 4

The smallest integer that satisfies this inequality is r s 10. The Chebyshev points corresponding to this value of r are z 0 , z1 , . . . , z10 , where by formula Ž9.18., z i s 4 q 4 cos

ž

2iq1 22

/

␲ ,

i s 0, 1, . . . , 10.

Using formula Ž9.37., the Lagrange interpolating polynomial that approximates hŽ x, ␪ . over w0, 8x is given by 10

Ý

p10 Ž x, ␪ . s ␪ 1

1 y ␪ 2 ey␪ 3 z i l i Ž x . ,

is0

where l i Ž x . is a polynomial of degree 10 which can be obtained from Ž9.13. by substituting z i for a i Ž i s 0, 1, . . . , 10.. 9.11. We have that

⭸ 4␩ Ž x, ␣ , ␤ . ⭸ x4

s Ž 0.49 y ␣ . ␤ 4 e 8 ␤ ey␤ x .

Hence, max 10FxF40

⭸ 4␩ Ž x, ␣ , ␤ . ⭸ x4

s Ž 0.49 y ␣ . ␤ 4 ey2 ␤ .

It can be verified that the function f Ž ␤ . s ␤ 4 ey2 ␤ is strictly monotone increasing for 0.06 F ␤ F 0.16. Therefore, ␤ s 0.16 maximizes f Ž ␤ .

627

CHAPTER 10

over this interval. Hence, max 10FxF40

⭸ 4␩ Ž x, ␣ , ␤ .

F Ž 0.49 y ␣ . Ž 0.16 . ey0 .32 4

⭸ x4

F 0.13 Ž 0.16 . ey0 .32 4

s 0.0000619, since 0.36 F ␣ F 0.41. Using Theorem 9.3.1, we have max 10FxF40

5 ␩ Ž x, ␣ , ␤ . y s Ž x . F 384 ⌬4 Ž 0.0000619 . .

Here, we considered equally spaced partition points with ⌬␶ i s ⌬. Let us now choose ⌬ such that 5 384

⌬4 Ž 0.0000619 . - 0.001.

This is satisfied if ⌬ - 5.93. Choosing ⌬ s 5, the number of knots needed is ms

40 y 10 ⌬

y1

s 5. 9.12. det Ž XX X . s Ž x 3 y ␣ . Ž x 2 y x 1 . y Ž x 3 y x 1 . Ž x 2 y ␣ . 2

2 2

.

The determinant is maximized when x 1 s y1, x 2 s 14 Ž1 q ␣ . 2 , x 3 s 1. CHAPTER 10 10.1. We have that 1

1

␲

␲

H cos nx cos mx dxs 2␲ Hy␲ ␲ y␲ s 1

½

0, 1, 1

␲

n / m, n s m G 1, ␲

H cos nx sin mx dxs 2␲ Hy␲ ␲ y␲ s0 1

␲

H sin nx sin mx dxs 2␲ Hy␲ ␲ y␲ s

½

sin Ž n q m . xy sin Ž n y m . x dx

for all m, n, 1

␲

cos Ž n q m . xq cos Ž n y m . x dx

0, 1,

cos Ž n y m . xy cos Ž n q m . x dx

n / m, n s m G 1.

628

SOLUTIONS TO SELECTED EXERCISES

10.2. (a)

(i) Integrating n times by parts w x m is differentiated and pnŽ x . is integrated x, we obtain

Hy1 x 1

m

pn Ž x . dxs 0

for m s 0, 1, . . . , n y 1.

(ii) Integrating n times by parts results in

Hy1 x 1

n

1

pn Ž x . dxs

2

Hy1 Ž 1 y x 1

n

2

n

. dx.

Letting xs cos ␪ , we obtain

Hy1 Ž 1 y x 1

2

␲

. dxs H sin 2 nq1␪ d␪ n

0

ž /

2 2 nq1

2n 2nq1 n

s

y1

n G 0.

,

(b) This is obvious, since Ža. is true for m s 0, 1, . . . , n y 1. (c)

Hy1 p 1

2 n

1

1

y1

2n

Ž x . dxs H

ž /

2n n x q ␲ny1 Ž x . pn Ž x . dx, n

where ␲ny1 Ž x . denotes a polynomial of degree n y 1. Hence, using the results in Ža. and Žb., we obtain

Hy1 p 1

2 n

Ž x . dxs s

10.3. (a)

1 2n

ž / 2

2nq1

½

Ž 2 i y 1. ␲ 2

n G 0.

,

Tn Ž ␨ i . s cos n Arccos cos s cos

ž /

nq1 2n 2 2n n 2nq1 n

Ž 2 i y 1. ␲ 2n

s0

for i s 1, 2, . . . , n. (b) TnX Ž x . s Ž nr '1 y x 2 . sinŽ n Arccos x .. Hence, TnX Ž ␨ i . s for i s 1, 2, . . . , n.

n

'1 y ␨

2 i

sin

Ž 2 i y 1. ␲ 2

/0

5

y1

629

CHAPTER 10

10.4. (a) dHn Ž x . dx

n

s Ž y1 . xe

x 2 r2

d n Ž eyx dx

2

.

r2

n

n

q Ž y1 . e

x 2 r2

d nq1 Ž eyx dx

2

r2

.

nq1

Using formulas Ž10.21. and Ž10.24., we have dHn Ž x . dx

n

s Ž y1 . xe x

r2

2

n

q Ž y1 . e x

2

n

Ž y1. eyx

r2

Ž y1.

2

r2

Hn Ž x .

nq1 yx 2 r2

e

Hnq1 Ž x .

s xHn Ž x . y Hnq1 Ž x . s nHny1 Ž x . ,

by formula Ž 10.25 . .

(b) From Ž10.23. and Ž10.24., we have Hnq1 Ž x . s xHn Ž x . y

dHn Ž x . dx

.

Hence, dHnq1 Ž x . dx d 2 Hn Ž x . dx

2

sx sx sx sx

dHn Ž x . dx dHn Ž x . dx dHn Ž x . dx dHn Ž x . dx

d 2 Hn Ž x .

q Hn Ž x . y

dx 2 dHnq1 Ž x .

q Hn Ž x . y

dx

q Hn Ž x . y Ž n q 1 . Hn Ž x . ,

by Ž a .

y nHn Ž x . ,

which gives the desired result. 10.5. (a) Using formula Ž10.18., we can show that sin Ž n q 1 . ␪ sin ␪

Fnq1

by mathematical induction. Obviously, the inequality is true for n s 1, since sin 2 ␪ sin ␪

s

2 sin ␪ cos ␪ sin ␪

F 2.

.

630

SOLUTIONS TO SELECTED EXERCISES

Suppose now that the inequality is true for n s m. To show that it is true for n s m q 1Ž m G 1.: sin Ž m q 2 . ␪ sin ␪

s

sin Ž m q 1 . ␪ cos ␪ q cos Ž m q 1 . ␪ sin ␪ sin ␪

F m q 1 q 1 s m q 2. Therefore, the inequality is true for all n. (b) From Section 10.4.2 we have that dTn Ž x . dx

sn

sin n␪ sin ␪

Hence, dTn Ž x . dx

sn

sin n␪ sin ␪

F n2 , since sin n␪rsin ␪ F n, which can be proved by induction as in Ža.. Note that as x™ .1, that is, as ␪ ™ 0 or ␪ ™ ␲ , dTn Ž x . dx

™ n2 .

(c) Making the change of variable t s cos ␪ , we get Tn Ž t .

Hy1 '1 y t x

2

H␲

Arccos x

dts y

1

Arccos x

sy sin n␪ n sy sy sy sy

1 n 1 n 1 n 1 n

cos n␪ d␪

␲

sin Ž n Arccos x . sin n ␺ ,

where xs cos ␺

sin ␺ Uny1 Ž x . ,

'1 y x 2 Uny1 Ž x . .

by Ž 10.18 .

631

CHAPTER 10

10.7. The first two Laguerre polynomials are L0 Ž x . s 1 and L1Ž x . s xy ␣ y 1, as can be seen from applying the Rodrigues formula in Section 10.6. Now, differentiating H Ž x, t . with respect to t, we obtain

⭸ H Ž x, t . ⭸t

␣q1

s

y

1yt

x

Ž1yt .

2

H Ž x, t . ,

or equivalently,

Ž1yt .

2

⭸ H Ž x, t . ⭸t

q Ž xy ␣ y 1 . q Ž ␣ q 1 . t H Ž x, t . s 0.

Hence, g 0 Ž x . s H Ž x, 0. s 1, and g 1Ž x . sy⭸ H Ž x, t . r⭸ t ts0 s x y ␣ y 1. Thus, g 0 Ž x . s L0 Ž x . and g 1Ž x . s L1Ž x .. Furthermore, if the representation

Ž y1.

⬁

Ý

H Ž x, t . s

n

n!

ns0

gnŽ x . t n

is substituted in the above equation, and if the coefficient of t n in the resulting series is equated to zero, we obtain g nq1 Ž x . q Ž 2 n y xq ␣ q 1 . g n Ž x . q Ž n2 q n ␣ . g ny1 Ž x . s 0. This is the same relation connecting the Laguerre polynomials L nq1 Ž x ., L nŽ x ., and L ny1 Ž x ., which is given at the end of Section 10.6. Since we have already established that g nŽ x . s L nŽ x . for n s 0, 1, we conclude that the same relation holds for all values of n. 10.8. Using formula Ž10.40., we have pU4

ž

Ž x . s c0 q c1 xq c2 q c3

ž

5x3

2 3x

y

2

/ / ž

3 x2

2

y

1

2

q c4

35 x 4 8

where c0 s c1 s c2 s

1 2 3 2 5

Hy1e 1

x

Hy1 xe 1

H 2 y1 1

ž

dx, x

dx,

3 x2 2

y

1 2

/

e x dx,

y

30 x 2 8

q

3 8

/

,

632

SOLUTIONS TO SELECTED EXERCISES

c3 s c4 s

7

H 2 y1 9

1

H 2 y1 1

ž ž

5x3

y

3x

2

2

35 x 4

y

/

e x dx,

30 x 2

8

q

8

3 8

/

e x dx.

In computing c 0 , c1 , c 2 , c 3 , c 4 we have made use of the fact that

Hy1 p 1

2 n

Ž x . dxs

2 2nq1

,

n s 0, 1, 2, . . . ,

where pnŽ x . is the Legendre polynomial of degree n Žsee Section 10.2.1.. 10.9. f Ž x. f

c0 2

q c1T1 Ž x . q c2 T2 Ž x . q c 3 T3 Ž x . q c 4 T4 Ž x .

s 1.266066 q 1.130318T1 Ž x . q 0.271495T2 Ž x . q 0.044337T3 Ž x . q 0.005474T4 Ž x . s 1.000044 q 0.99731 xq 0.4992 x 2 q 0.177344 x 3 q 0.043792 x 4 w Note: For more computational details, see Example 7.2 in Ralston and Rabinowitz Ž1978..x 10.10. Use formula Ž10.48. with the given values of the central moments. 10.11. The first six cumulants of the standardized chi-squared distribution with five degrees of freedom are ␬ 1 s 0, ␬ 2 s 1, ␬ 3 s 1.264911064, ␬ 4 s 2.40, ␬ 5 s 6.0715731, ␬ 6 s 19.2. We also have that z 0.05 s 1.645. Applying the Cornish-Fisher approximation for x 0.05 , we obtain the value x 0.05 f 1.921. Thus, P Ž ␹ 5U 2 ) 1.921. f 0.05, where ␹ 5U 2 denotes the standardized chi-squared variate with five degrees of freedom. If ␹ 52 denotes the nonstandardized chi-squared counterpart Žwith five degrees of freedom., then ␹ 52 s '10 ␹ 5U 2 q 5. Hence, the corresponding approximate value of the upper 0.05 quantile of ␹ 52 is '10 Ž1.921. q 5 s 11.0747. The actual table value is 11.07. 10.12. (a)

H0 e

1 yt 2 r2

dtf 1 y q

1 2 = 3 = 1! 1

2 = 9 = 4! 4

q

1 2 = 5 = 2! 2

s 0.85564649.

y

1 2 = 7 = 3! 3

633

CHAPTER 10

(b)

H0 e x

yt 2 r2

dtf xeyx

r8

2

⌰0

ž / x

q

2

1 3

⌰2

ž / x

2

ž /

ž /

1 x 1 x q ⌰4 q ⌰6 5 2 7 2

,

where ⌰0

ž / x

s

2

1 1!

H0

ž /

ž ž ž ž

/ / / /

x

2

s 1, ⌰2

ž / x

s

2

1

x

2! 2 s

1

x

2! 2 ⌰4 ⌰6

ž / ž / x

s

2 x

1

x

4! 2 s

2

1

x

6! 2

ž / ž / ž / ž / ž / ž /

2

H2 2

x

2

2

x

y1 ,

2

4

4

x

2

6

2

x

y6

2

6

x

x

y 15

2

q3 , 4

q 45

2

ž / x

2

2

y 15 .

Hence,

H0 e

1 yt 2 r2

dtf 0.85562427.

10.13. Using formula Ž10.21., we have that gŽ x. s

⬁

Ý bn Ž y1.

n

e

x 2 r2

d n Ž eyx dx

ns0

s

⬁

Ý Ž y1.

n

bn

ns0

s

⬁

Ý

ns0

'2␲

c n d n␾ Ž x . n!

dx n

1

,

where c n s Žy1. n n!bn , n s 0, 1, . . .

2

r2

.

n

d n Ž eyx dx

2

n

r2

.

␾Ž x.

,

by Ž 10.44 .

634

SOLUTIONS TO SELECTED EXERCISES

10.14. The moment generating function of any one of the X i2 ’s is

␺ Ž t. s s

⬁

Hy⬁e ⬁

Hy⬁

tx2

f Ž x . dx

et x ␾ Ž x . y 2

␭3 d 3␾ Ž x . dx 3

6

q

␭4 d 4␾ Ž x .

q

dx 4

24

␭23 d 6␾ Ž x . 72

dx 6

dx.

By formula Ž10.21., d n␾ Ž x . dx

n

s Ž y1 . ␾ Ž x . Hn Ž x . ,

n

n s 0, 1, 2, . . .

Hence,

␺ Ž t. s

⬁

Hy⬁e

tx2

␾Ž x. q

␭3

␾ Ž x . H3 Ž x . q

6

q

␭4 24

␭23 72

⬁

H0

s2

␭4

et x ␾ Ž x . q 2

24

␾ Ž x . H4 Ž x . q

␾ Ž x . H4 Ž x . ␾ Ž x . H6 Ž x . dx

␭23 72

␾ Ž x . H6 Ž x . dx,

since H3 Ž x . is an odd function, and H4Ž x . and H6 Ž x . are even functions. It is known that ⬁

H0 e

x 2 ny1 dxs 2 ny1 ⌫ Ž n . ,

yx 2 r2

where ⌫ Ž n. is the gamma function ⌫ Ž n. s

⬁

H0 e

yx

⬁

H0 e

x ny1 dxs 2

yx 2 2 ny1

x

dx,

n ) 0. It is easy to show that ⬁

H0 e

tx2

␾ Ž x . x m dxs s

1

'2␲ 2

m 2

2'␲

⬁

H0 e

yŽ x 2 r2. Ž1y2 t .

Ž1y2 t .

y 21 Ž mq1 .

x m dx ⌫

ž

mq1 2

/

,

635

CHAPTER 11

where m is an even integer. Hence, ⬁

H0 e

2

tx2

␾ Ž x . dxs

1

y1r2 ⌫ Ž 12 . '␲ Ž 1 y 2 t .

s Ž1y2 t . ⬁

H0 e

2

tx2

⬁

H0 e

␾ Ž x . H4 Ž x . dxs 2 s

⬁

H0 e

tx2

␾ Ž x . Ž x 4 y 6 x 2 q 3 . dx

y5r2 ⌫ Ž 52 . '␲ Ž 1 y 2 t .

12

y3r2 y1r2 ⌫ Ž 32 . q 3 Ž 1 y 2 t . , '␲ Ž 1 y 2 t .

⬁

H0 e

␾ Ž x . H6 Ž x . dxs 2 s

,

4

y 2

tx2

y1r2

8

'␲ q

tx2

␾ Ž x . Ž x 6 y 15 x 4 q 45 x 2 y 15 . dx

Ž1y2 t .

y7r2

⌫ Ž 72 . y

60

y5r2 ⌫ Ž 52 . '␲ Ž 1 y 2 t .

90

y3r2 y1r2 ⌫ Ž 32 . y 15 Ž 1 y 2 t . . '␲ Ž 1 y 2 t .

The last three integrals can be substituted in the formula for ␺ Ž t ..The moment generating function of W is w ␺ Ž t .x n. On the other hand, the moment generating function of a chi-squared distribution with n degrees of freedom is Ž1 y 2 t .yn r2 wsee Example 6.9.8 concerning the moment generating function of a gamma distribution GŽ ␣ , ␤ ., of which the chi-squared distribution is a special case with ␣ s nr␣ , ␤ s 2x. CHAPTER 11 11.1. (a) a0 s s an s s

1

␲

2

␲

1

␲

2

␲

H ␲ y␲

x dx

H x dxs ␲ , ␲ 0 H ␲ y␲

x cos nx dx

H xcos nx dx ␲ 0

636

SOLUTIONS TO SELECTED EXERCISES

sy

s

bn s

2

␲

H sin nx dx n␲ 0

2

␲n 1

n

2

Ž y1. y 1 ,

␲

H ␲ y␲

x sin nx dx

s 0, x s

␲

y

2

4

␲

ž

cos 3 x

cos xq

3

2

q

cos 5 x 52

/

q ⭈⭈⭈ .

(b) a0 s s

an s s

s

1

␲

2

␲

1

␲

2

␲

1

␲

H sin x ␲ y␲

dx 4

H sin x dxs ␲ , ␲ 0 H sin x cos nx dx ␲ y␲ H sin x cos nx dx ␲ 0 H ␲ 0

sin Ž n q 1 . xy sin Ž n y 1 . x dx n

Ž y1. q 1 s y2 , ␲ Ž n2 y 1 .

n / 1,

a1 s 0, bn s

1

␲

H sin x ␲ y␲

sin nx dx

s 0, sin x s

2

␲

y

4

␲

ž

cos 2 x 3

q

cos 4 x 15

q

cos 6 x 35

/

q ⭈⭈⭈ .

637

CHAPTER 11

(c) a0 s s

1

␲

H Ž xq x ␲ y␲ 2␲ 2

s s bn s s

1

␲

2

␲

H Ž xq x ␲ y␲ Hx ␲ 0 4 n

2

␲2 3

2

. cos nx dx

2

cos nx dx

cos n␲ s Ž y1 .

1

␲

2

␲

H Ž xq x ␲ y␲

2

n

4 n2

,

. sin nx dx

H x sin nx dx ␲ 0

s Ž y1 . xq x 2 s

. dx

,

3 an s

2

2

ny1

n

,

ž

q 4 ycos xq

ž

1

y4 y

2

cos 2 xq

2

1 2 1 4

sin x

/

/

sin 2 x q ⭈⭈⭈

for y␲ - x- ␲ . When xs .␲ , the sum of the series is 12 wŽy␲ q ␲ 2 . q Ž␲ q ␲ 2 .x s ␲ 2 , that is, ␲ 2 s ␲ 2r3 q 4Ž1 q 1r2 2 q 1r3 2 q ⭈⭈⭈ .. 11.2. From Example 11.2.2, we have x s 2

␲2 3

q4

⬁

Ý

Ž y1. n2

ns1

Putting xs .␲ , we get

␲2 6

s

⬁

1

ns1

n2

Ý

.

n

cos nx.

638

SOLUTIONS TO SELECTED EXERCISES

Using xs 0, we obtain

␲2

0s

3

⬁

q4

Ý

Ž y1.

n

,

n2

ns1

or equivalently,

␲2

⬁

Ý

s

12

Ž y1.

nq1

.

n2

ns1

Adding the two series corresponding to ␲ 2r6 and ␲ 2r12, we get 3␲ 2 12

⬁

1

ns1

Ž 2 n y 1.

Ý

s2

2

,

that is,

␲2

⬁

1

ns1

Ž 2 n y 1.

s

8

Ý

2

.

11.3. By Theorem 11.3.1, we have ⬁

Ý Ž nbn cos nxy nan sin nx . .

fXŽ x. s

ns1

Furthermore, inequality Ž11.28. indicates that Ý⬁ns1 Ž n2 bn2 q n2 a2n . is a convergent series. It follows that nbn ™ 0 and nan ™ 0 as n ™ ⬁ Žsee Result 5.2.1 in Section 5.2.. 11.4. For m ) n, we have m

Ý Ž ak cos kxq bk sin kx .

sm Ž x . y sn Ž x . s

ksnq1 m

Ý Ž a2k q bk2 .

F

1r2

ksnq1 m

Ý

s

ksnq1

F

ž

1

Ž ␣ 2 q ␤k2 . k k

m

Ý

ksnq1

1 k2

/

1r2

1r2

see Ž 11.27 .

m

Ý Ž

ksnq1

␣ k2 q ␤ k2

.

1r2

.

639

CHAPTER 11

But, by inequality Ž11.28., m

1

␲

Ý Ž ␣ k2 q ␤k2 . F ␲ H

fXŽ x.

y␲

ksnq1

2

dx

and m

1

Ý

ksnq1

k

⬁

1

ksnq1

k2

Ý

F

2

F

dx

⬁

Hn

x

2

s

1

.

n

Thus, c

sm Ž x . y sn Ž x . F

,

'n

where c2 s

1

␲

H ␲ y␲

fXŽ x.

2

dx.

By Theorem 11.3.1Žb., smŽ x . ™ f Ž x . on wy␲ , ␲ x as m ™ ⬁. Thus, by letting m ™ ⬁, we get f Ž x . y sn Ž x . F

c

.

'n

11.5. (a) From the proof of Theorem 11.3.2, we have ⬁

Ý

ns1

bn n

A0

cos n␲ s

y

a0 ␲

2

2

.

This implies that the series Ý⬁ns1 Žy1. n bnrn is convergent. (b) From the proof of Theorem 11.3.2, we have

Hy␲ f Ž t . dts x

a0 Ž ␲ q x .

q

2

⬁

Ý

an n

ns1

sin nxy

bn n

Ž cos nxy cos n␲ . .

Putting xs 0, we get

Hy␲ f Ž t . dts 0

a0 ␲ 2

y

⬁

Ý

ns1

bn n

1 y Ž y1 .

n

.

This implies convergence of the series Ý⬁ns1 Ž bnrn.w1 y Žy1. n x. Since Ý⬁ns1 Žy1. n bnrn is convergent by part Ža., then so is Ý⬁ns1 bnrn.

640

SOLUTIONS TO SELECTED EXERCISES

11.6. Using the hint and part Žb. of Exercise 11.5, the series Ý⬁ns2 bnrn would be convergent, where bn s 1rlog n. However, the series Ý⬁ns2 1rŽ n log n. is divergent by Maclaurin’s integral test Žsee Theorem 6.5.4.. 11.7. (a) From Example 11.2.1, we have x

s

2

Ž y1.

⬁

Ý

nq1

sin nx

n

ns1

for y␲ - x- ␲ . Hence, x2

s

4

H0

x

t

dt

2

Ž y1.

⬁

Ý

sy

nq1

⬁

cos nxq

n2

ns1

Ý

Ž y1. n2

ns1

nq1

.

Note that ⬁

Ý

Ž y1.

nq1

s

n2

ns1

s

1

␲

x2

H 2␲ y␲ 4 ␲2 12

dx

.

Hence, x2

s

4

␲2

y

12

⬁

Ý

Ž y1.

nq1

cos nx,

n2

ns1

y␲ - x- ␲ .

(b) This follows directly from part Ža.. 11.8. Using the result in Exercise 11.7, we obtain

H0

x

ž

␲2 12

y

t2 4

/

⬁

dts

Ý

Ž y1. n

ns1

s

⬁

Ý

2

Ž y1.

ns1

nq1

H0 cos nt dt x

nq1

n3

sin nx,

y␲ - x- ␲ .

Hence, ⬁

Ý

ns1

Ž y1. n3

nq1

sin nxs

␲2 12

xy

x3 12

,

y␲ - x- ␲ .

641

CHAPTER 11

11.9.

1

⬁

i

⬁

H e 2␲ y⬁

FŽ w. s

yx 2 yi w x

e

dx,

H Ž y2 xe 4␲ y⬁

FX Ž w . s

yx 2

. eyi w x dx

Žexchanging the order of integration and differentation is permissible here by an extension of Theorem 7.10.1.. Integrating by parts, we obtain FX Ž w . s s

i

2

4␲ yi

⬁

yx 2

w 1

⬁

H e 4␲ y⬁

sy sy

⬁ y⬁ y

eyx eyi w x

2 w 2

⬁

Hy⬁e

yx 2

Ž yiweyi w x . dx

Ž yiw . eyi w x dx

H e 2␲ y⬁

yx 2 yi w x

e

dx

FŽ w. .

The general solution of this differential equation is F Ž w . s Aeyw

2

r4

,

where A is a constant. Putting w s 0, we obtain A s F Ž 0. s s

'␲ 2␲

Hence, F Ž w . s Ž1r2'␲ . eyw

2

s

r4

1

⬁

H e 2␲ y⬁ 1

yx 2

dx

.

2'␲

.

11.10. Let H Ž w . be the Fourier transform of Ž f ) g .Ž x .. Then HŽ w. s

1

⬁

1

⬁

⬁

H H f Ž xy y . g Ž y . dy 2␲ y⬁ y⬁ ⬁

s

H g Ž y . Hy⬁ f Ž xy y . e 2␲ y⬁

s

Hy⬁

⬁

sFŽ w.

1

yi w x

⬁

H f Ž xy y . e 2␲ y⬁ ⬁

yi wŽ xyy .

Hy⬁ g Ž y . e

s 2␲ F Ž w . G Ž w . .

yi w y

eyi w x dx

dy

dx dy

dx g Ž y . eyi w y dy

642

SOLUTIONS TO SELECTED EXERCISES

11.11. Apply the results in Exercises 11.9 and 11.10 to find the Fourier transform of f Ž x .; then apply the inverse Fourier transform theorem ŽTheorem 11.6.3. to find f Ž x .. 11.12. (a) sn Ž x n . s

n

Ý

2 Ž y1 .

n

s

Ý

2 Ž y1 .

n

Ý

Ž y1.

2 sin Ž k␲rn . k

ks1

␲ n

kq1

k

ks1

s

sin k ␲ y

k

ks1

ž

kq1

kq1

sin

/

ž / k␲ n

.

(b) sin Ž k␲rn . ␲

n

sn Ž x n . s 2

Ý

k␲rn

ks1

n

.

By dividing the interval w0, ␲ x into n subintervals wŽ k y 1.␲rn, k␲rn x, 1 F k F n, each of length ␲rn, it is easy to see that snŽ x n . is a Riemann sum S Ž P, g . for the function g Ž x . s Ž2 sin x .rx. Here, g Ž x . is evaluated at the right-hand end point of each subinterval of the partition P Žsee Section 6.1.. Thus, ␲

H0

lim sn Ž x n . s 2

n™⬁

sin x x

dx.

11.13. (a) ˆ y s 8.512 q 3.198 cos ␾ y 0.922 sin ␾ q 1.903 cos 2 ␾ q 3.017 sin 2 ␾ . (b) Estimates of the locations of minimum and maximum resistance are ␾ s 0.7944␲ and ␾ s 0.1153␲ , respectively. w Note: For more details, see Kupper Ž1972., Section 5.x 11.15. (a) sUn s

s

1

␴'n 1

␴'n

ž

n

Ý Yj y n ␮

js1 n

Ý Uj ,

js1

/

643

CHAPTER 11

where Uj s Yj y ␮ , j s 1, 2, . . . , n. Note that E ŽUj . s 0 and VarŽUj . s ␴ 2 . The characteristic function of sUn is U

␾n Ž t . s E Ž e it s n . s E Ž e it Ý js1 U j r ␴'n . n

n

Ł E Ž e i t U r ␴'n . .

s

j

js1

Now,

ž

E Ž e it U j . s E 1 q it Uj y s1y EŽ e

it U j r ␴ 'n

. s1y

t 2␴ 2 2 t2 2n

t2 2

Uj 2 q ⭈⭈⭈

/

q oŽ t 2 . ,

qo

ž /

qo

ž /

t2 n

.

Hence,

␾n Ž t . s 1 y

t2 2n

t2

n

.

n

Let

␻n s y s

t2 n

t2 2n y

qo 1 2

ž / t2 n

q o Ž 1. ,

␾n Ž t . s Ž 1 q ␻ n . t

2

r␻ n

s Ž 1 q ␻n . t

2

r␻ n

y1r2qo Ž1 .

y1r2

Ž 1 q ␻ n . t r␻ n 2

As n ™ ⬁, ␻ n ™ 0 and

Ž 1 q ␻ n . t r␻ n 2

y1r2

Ž 1 q ␻ n . t r␻ n 2

o Ž1 .

™ eyt ™ 1.

2

r2

,

o Ž1 .

.

644

SOLUTIONS TO SELECTED EXERCISES

Thus,

␾n Ž t . ™ eyt

2

r2

as n ™ ⬁.

(b) eyt r2 is the characteristic function of the standard normal distribution Z. Hence, by Theorem 11.7.3, as n ™ ⬁, sUn ™ Z. 2

CHAPTER 12 12.1. (a)

In f S n s

h 2

s

ž

ny1

log 1 q log n q 2

ny1 2 Ž n y 1.

ž

Ý log x i

is1

ny1

log n q 2

Ý log i

is2

/

/

s 12 log n q log 2 q log 3 q ⭈⭈⭈ qlog Ž n y 1 . s 12 log n qlog Ž n! . y log n s log Ž n! . y 12 log n. (b) n log n y n q 1 f logŽ n!. y 12 log n. Hence, log Ž n! . f Ž n q 12 . log n y n q 1

Ž n q 12 . log n y n q 1

n!f exp

s e eyn n nq1r2 s eynq1 n nq1r2 . 12.2. Applying formula Ž12.10., we obtain the following approximate values for the integral H01dxrŽ1 q x .: n

Approximation

2 4 8 16

0.69325395 0.69315450 0.69314759 0.69314708

The exact value of the integral is log 2 s 0.69314718. Using formula Ž12.12., the error of approximation is less than or equal to M4 Ž b y a . 2880n

4

5

s

M4 2880 Ž 8 .

4

,

645

CHAPTER 12

where M4 is an upper bound on f Ž4. Ž x . for 0 F xF 1. Here, f Ž x . s 1rŽ1 q x ., and 24

f Ž4. Ž x . s

Ž1qx.

5

.

Hence, M4 s 24, and M4 2880 Ž 8 . 12.3. Let zs

␪s

f 0.000002.

4

2 ␪ y ␲r2

␲r2 ␲ 4

0F␪F

,

Ž1qz. ,

␲ 2

,

y1 F z F 1.

Then

H0

␲r2

sin ␪ d␪ s f

␲

H sin 4 y1

␲ 4

1

␲ 4

2

Ý ␻ i sin

is0

Ž 1 q z . dz ␲ 4

Ž 1 q zi . .

Using the information in Example 12.4.1, we have z 0 s y z2 s

'

3 5

; ␻ 0 s , ␻ 1 s , ␻ 2 s . Hence, 5 9

H0

␲r2

8 9

sin ␪ d␪ f

5 9

␲ 4

½

5 9

␲

sin

4

'

3 5

, z1 s 0,

ž (/ ž ( /5 1y

3

q

5

5 ␲ q sin 1q 9 4

8

9

sin

␲ 4

3

5

s 1.0000081. 1 The error of approximation associated with Hy1 sinwŽ␲r4.Ž1 q z .x dz is obtained by applying formula Ž12.16. with as y1, bs 1, f Ž z . s sinwŽ␲r4.Ž1 q z .x, n s 2. Thus, f Ž6. Ž ␰ . s yŽ␲r4. 6 sinwŽ␲r4.Ž1 q ␰ .x, y1 - ␰ - 1, and the error is therefore less than

2 7 Ž 3! . 7 Ž 6! .

4

3

ž / ␲ 4

6

.

646

SOLUTIONS TO SELECTED EXERCISES

Consequently, the error associated with H0␲ r2 sin ␪ d␪ is less than

␲ 2 7 w 3! x

4

4 7 w 6! x 3

ž / ž / ␲

␲

6

s

4

7

2

w 3! x 4 3 7 w 6! x

s 0.0000117. 12.4. (a) The inequality is obvious from the hint. (b) Let m s 3. Then, 1 m

eym s 2

1 3

ey9

s 0.0000411. (c) Apply Simpson’s method on w0, 3x with h - 0.2. Here, f Ž4. Ž x . s Ž 12 y 48 x 2 q 16 x 4 . eyx , 2

with a maximum of 12 at xs 0 over w0, 3x. Hence, M4 s 12. Using the fact that h s Ž by a.r2 n, we get n s Ž by a.r2 h ) 3r0.4 s 7.5. Formula Ž12.12., with n s 8, gives nM4 h 5 90

-

8 Ž 12 . Ž 0.2 .

5

90

s 0.00034. Hence, the total error of approximation for H0⬁ eyx dx is less than 0.0000411 q 0.00034 s 0.00038. This makes the approximation correct to three decimal places. 2

12.5. x

⬁

H0

e x q eyx y 1

dxs

x eyx

⬁

H0

1 q ey2 x y eyx

n

f

dx,

xi

Ý ␻ i 1 q ey2 x y eyx i

is0

i

,

where the x i ’s are the zeros of the Laguerre polynomial L nq1 Ž x .. Using the entries in Table 12.1 for n s 1, we get x

⬁

H0

yx

e qe x

y1

dxf 0.85355

0.58579 y2Ž 0.58579.

1qe

q0.14645

y ey0 .58579

3.41421 y2Ž 3.41421.

1qe

y ey3 .41421

s 1.18.

647

CHAPTER 12

12.6. (a) Let u s Ž2 t y x .rx. Then IŽ x. s

1

H 2 y1

x

1

1 q x 2 Ž u q 1. 1 4

Hy1 4 q x

s2 x

1

1

2

Ž u q 1.

2

2

du

du.

(b) Applying a Gauss᎐Legendre approximation of I Ž x . with n s 4, we obtain

␻i

4

IŽ x. f2 x Ý

4 q x Ž u i q 1. 2

is0

2

,

where u 0 , u1 , u 2 , u 3 , and u 4 are the five zeros of the Legendre polynomial P5 Ž u. of degree 5. Values of these zeros and the corresponding ␻ i ’s are given below: u 0 s 0.90617985,

␻ 0 s 0.23692689,

u1 s 0.53846931,

␻ 1 s 0.47862867,

u 2 s 0,

␻ 2 s 0.56888889,

u 3 s y0.53846931,

␻ 3 s 0.47862867,

u 4 s y0.90617985,

␻4 s 0.23692689,

Žsee Table 7.5 in Shoup, 1984, page 202.. 12.7. H01 Žcos x . ␭ dxs H01 e ␭ log cos x dx. Using the method of Laplace, the function hŽ x . s log cos x has a single maximum in w0, 1x at xs 0. Formula Ž12.28. gives

H0 e

1 ␭ log cos x

dx;

y␲

1r2

2 ␭ hY Ž 0 .

as ␭ ™ ⬁, where hY Ž 0 . sy

1 cos 2 x

xs0

s y1. Hence,

H0 e

1 ␭ log cos x

dx;

(

␲ 2␭

.

648 12.9.

SOLUTIONS TO SELECTED EXERCISES

1 H0 H0Ž1yx . Ž1 y x 2 1r2 1

.

2 2 1r2 1 y x2

dx 1 dx 2 s

␲ 6

f 0.52359878. Applying the 16-point Gauss᎐Legendre rule to both formulas gives the approximate value 0.52362038. w Note: For more details, see Stroud Ž1971, Section 1.6..x 12.10.

⬁

H0

Ž1qx2 .

yn

dxs

⬁

H0 e

yn log Ž1qx 2 .

dx

Apply the method of Laplace with ␸ Ž x . s 1 and h Ž x . s ylog Ž 1 q x 2 . . This function has a single maximum in w0, ⬁. at xs 0. Using formula Ž12.28., we get ⬁

H0

Ž1qx2 .

dx;

where h⬙ Ž0. s y2. Hence, ⬁

H0

Ž1qx2 .

y␲

yn

yn

dx;

(

1r2

,

2 n hY Ž 0 .

␲ 4n

s

1 2

(

␲ n

.

12.11. Number of Quadrature Points

Approximate Value of Variance

2 4 8 16 32 64

0.1175 0.2380 0.2341 0.1612 0.1379 0.1372

w Source: Example 16.6.1 in Lange Ž1999..x 12.12. If xG 0, then ⌽Ž x. s s

1

'2␲ Hy⬁e 1 2

q

0

yt 2 r2

1

H0 e

'2␲

x

dtq

yt 2 r2

1

'2␲ H0 e

dt.

x

yt 2 r2

dt

649

CHAPTER 12

Using Maclaurin’s series Žsee Section 4.3., we obtain ⌽Ž x. s

1

q

2

1

'2␲

H0

x

1q

q

1

q

2

1

'2␲

ty

t2

y

1!

1 3!

s

ž / ž / ž / ž /

1

q

2

y

1

q

2

x

'2␲

1y

y

2!

1

q ⭈⭈⭈ q

2

t3

q

3 = 2 = 1!

x2 3 = 2 = 1!

q

q ⭈⭈⭈ q Ž y1 .

2

t2 2

3

t2

q ⭈⭈⭈ q Ž y1 .

s

1

y

n!

t2 2

t5 5 = 2 2 = 2!

y

n

q ⭈⭈⭈ dt t7

7 = 2 3 = 3! x

t 2 nq1

n

Ž 2 n q 1 . = 2 n = n! x4 5 = 2 2 = 2!

y

0

x6 7 = 2 3 = 3!

x2n

n

q ⭈⭈⭈

Ž 2 n q 1 . = 2 n = n!

q ⭈⭈⭈ .

12.13. From Exercise 12.12 we need to find values of a, b, c, d such that

ž

1 q ax 2 q bx 4 f Ž 1 q cx 2 q dx 4 . 1 y

x2

q

6

x4

y

40

x6 336

x8

q

3456

/

.

Equating the coefficients of x 2 , x 4 , x 6 , x 8 on both sides yields four equations in a, b, c, d. Solving these equations, we get as

17 468

,

bs

739 196560

cs

,

95 468

,

ds

55 4368

.

w Note: For more details, see Morland Ž1998..x 12.14. ysGŽ x. s s

1 4

ž

xq

H0 x2 2

x

1 4

/

Ž 1 q t . dt

,

0 F xF 2.

The only solution of y s GŽ x . in w0, 2x is x s y1 q Ž1 q 8 y .1r2 , 0 F y

650

SOLUTIONS TO SELECTED EXERCISES

F 1. A random sample of size 150 from the g Ž x . distribution is given by x i s y1 q Ž 1 q 8 yi .

1r2

i s 1, 2, . . . , 150,

,

where the yi ’s form a random sample from the U Ž0, 1. distribution. Applying formula Ž12.43., we get

U I150 s

ˆ

2

e xi

150

4

Ý

150

is1

1 q xi

s 16.6572. Using now formula Ž12.36., we obtain

Iˆ150 s

150

2 150

Ý eu

2 i

is1

s 17.8878, where the u i ’s form a random sample from the U Ž0, 2. distribution. 12.15. As n ™ ⬁,

1q

x2

y Ž nq1 .r2

s 1q

n

; eyx

2

x2

ynr2

n

r2

1q

x2

y1r2

n

.

Furthermore, by applying Stirling’s formula Žformula 12.31., we obtain

⌫

ž

nq1 2 ⌫

/

ž / n 2

; ey

1 2

Ž ny1 .

y 12 Ž ny2 .

;e

ny1

Ž ny1 .r2

2␲

2 ny2 2

Ž ny2 .r2

2␲

ž ž

ny1 2 ny2 2

/ /

1r2

, 1r2

.

651

CHAPTER 12

Hence, ⌫

ž

nq1 2

'n␲ ⌫

/

ž / n

;

1

ny1

e1r2

ny2

1

'n␲

;

; s

'n␲

1 e

1

'n␲ 1

'2␲

1r2

1 e

1r2

(

2

2

1

ny2

nr2

Ž 1 y n1 . nr2 Ž 1 y n2 . nr2

e

(

1

x

ey1r2 y1

ny1 2

ny2

(

2

ny1

n 2

.

Hence, for large n, FŽ x. f

'2␲ Hy⬁e

yt 2 r2

dt.

2

ny1 ny2

1r2

General Bibliography

Abramowitz, M., and I. A. Stegun Ž1964.. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Wiley, New York. Abramowitz, M., and I. A. Stegun, eds. Ž1972.. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Wiley, New York. Adby, P. R., and M. A. H. Dempster Ž1974.. Introduction to Optimization Methods. Chapman and Hall, London. Agarwal, G. G., and W. J. Studden Ž1978.. ‘‘Asymptotic design and estimation using linear splines.’’ Comm. Statist. Simul. Comput., 7, 309᎐319. Alvo, M., and P. Cabilio Ž2000.. ‘‘Calculation of hypergeometric probabilities using Chebyshev polynomials.’’ Amer. Statist., 54, 141᎐144. Anderson, T. W., and S. D. Gupta Ž1963.. ‘‘Some inequalities on characteristic roots of matrices.’’ Biometrika, 50, 522᎐524. Anderson, T. W., I. Olkin, and L. G. Underhill Ž1987.. ‘‘Generation of random orthogonal matrices.’’ SIAM J. Sci. Statist. Comput., 8, 625᎐629. Anderson-Cook, C. M. Ž2000.. ‘‘A second order model for cylindrical data.’’ J. Statist. Comput. Simul., 66, 51᎐56. Apostol, T. M. Ž1964.. Mathematical Analysis. Addison-Wesley, Reading, Massachusetts. Ash, A., and A. Hedayat Ž1978.. ‘‘An introduction to design optimality with an overview of the literature.’’ Comm. Statist. Theory Methods, 7, 1295᎐1325. Atkinson, A. C. Ž1982.. ‘‘Developments in the design of experiments.’’ Internat. Statist. Re®., 50. 161᎐177. Atkinson, A. C. Ž1988.. ‘‘Recent developments in the methods of optimum and related experimental designs.’’ Internat. Statist. Re®., 56, 99᎐115. Basilevsky, A. Ž1983.. Applied Matrix Algebra in the Statistical Sciences. North-Holland, New York. Bates, D. M., and D. G. Watts Ž1988.. Nonlinear Regression Analysis and its Applications. Wiley, New York. Bayne, C. K., and I. B. Rubin Ž1986.. Practical Experimental Designs and Optimization Methods for Chemists. VCH Publishers, Deerfield Beach, Florida. Bellman, R. Ž1970.. Introduction to Matrix Analysis, 2nd ed. McGraw-Hill, New York.

652

GENERAL BIBLIOGRAPHY

653

Belsley, D. A., E. Kuh, and R. E. Welsch Ž1980.. Regression Diagnostics. Wiley, New York. Bickel, P. J., and K. A. Doksum Ž1977.. Mathematical Statistics. Holden-Day, San Francisco. Biles, W. E., and J. J. Swain Ž1980.. Optimization and Industrial Experimentation. Wiley-Interscience, New York. Bloomfield, P. Ž1976.. Fourier Analysis of Times Series: An Introduction. Wiley, New York. Blyth, C. R. Ž1990.. ‘‘Minimizing the sum of absolute deviations.’’ Amer. Statist., 44, 329. Bohachevsky, I. O., M. E. Johnson, and M. L. Stein Ž1986.. ‘‘Generalized simulated annealing for function optimization.’’ Technometrics, 28, 209᎐217. Box, G. E. P. Ž1982.. ‘‘Choice of response surface design and alphabetic optimality.’’ Utilitas Math., 21B, 11᎐55. Box, G. E. P., and D. W. Behnken Ž1960.. ‘‘Some new three level designs for the study of quantitative variables.’’ Technometrics, 2, 455᎐475. Box, G. E. P., and D. R. Cox Ž1964.. ‘‘An analysis of transformations.’’ J. Roy. Statist. Soc. Ser. B, 26, 211᎐243. Box, G. E. P., and N. R. Draper Ž1959.. ‘‘A basis for the selection of a response surface design.’’ J. Amer. Statist. Assoc., 54, 622᎐654. Box, G. E. P., and N. R. Draper Ž1963.. ‘‘The choice of a second order rotatable design.’’ Biometrika, 50, 335᎐352. Box, G. E. P., and N. R. Draper Ž1965.. ‘‘The Bayesian estimation of common parameters from several responses.’’ Biometrika, 52, 355᎐365. Box, G. E. P., and N. R. Draper Ž1987.. Empirical Model-Building and Response Surfaces. Wiley, New York. Box, G. E. P., and H. L. Lucas Ž1959.. ‘‘Design of experiments in nonlinear situations.’’ Biometrika, 46, 77᎐90. Box, G. E. P., and K. B. Wilson Ž1951.. ‘‘On the experimental attainment of optimum conditions.’’ J. Roy. Statist. Soc. Ser. B, 13, 1᎐45. Box, G. E. P., W. G. Hunter, and J. S. Hunter Ž1978.. Statistics for Experimenters. Wiley, New York. Boyer, C. B. Ž1968.. A History of Mathematics. Wiley, New York. Bronshtein, I. N., and K. A. Semendyayev Ž1985.. Handbook of Mathematics. ŽEnglish translation edited by K. A. Hirsch.. Van Nostrand Reinhold, New York. Brownlee, K. A. Ž1965.. Statistical Theory and Methodology, 2nd ed. Wiley, New York. Buck, R. C. Ž1956.. Ad®anced Calculus. McGraw-Hill, New York. Bunday, B. D. Ž1984.. Basic Optimization Methods. Edward Arnold, Victoria, Australia. Buse, A., and L. Lim Ž1977.. ‘‘Cubic splines as a special case of restricted least squares.’’ J. Amer. Statist. Assoc., 72, 64᎐68. Bush, K. A., and I. Olkin Ž1959.. ‘‘Extrema of quadratic forms with applications to statistics.’’ Biometrika, 46, 483᎐486.

654

GENERAL BIBLIOGRAPHY

Carslaw, H. S. Ž1930.. Introduction to the Theory of Fourier Series and Integrals, 3rd ed. Dover, New York. Chatterjee, S., and B. Price Ž1977.. Regression Analysis by Example. Wiley, New York. Cheney, E. W. Ž1982.. Introduction to Approximation Theory, 2nd ed. Chelsea, New York. Chihara, T. S. Ž1978.. An Introduction to Orthogonal Polynomials. Gordon and Breach, New York. Churchill, R. V. Ž1963.. Fourier Series and Boundary Value Problems, 2nd ed. McGraw-Hill, New York. Cochran, W. G. Ž1963.. Sampling Techniques, 2nd ed. Wiley, New York. Conlon, M. Ž1991.. ‘‘The controlled random search procedure for function optimization.’’ Personal communication. Conlon, M., and A. I. Khuri Ž1992.. ‘‘Multiple response optimization.’’ Technical Report, Department of Statistics, University of Florida, Gainesville, Florida. Cook, R. D., and C. J. Nachtsheim Ž1980.. ‘‘A comparison of algorithms for constructing exact D-optimal designs.’’ Technometrics, 22, 315᎐324. Cooke, W. P. Ž1988.. ‘‘L’Hopital’s rule in a Poisson derivation.’’ Amer. Math. Monthly, ˆ 95, 253᎐254. Copson, E. T. Ž1965.. Asymptotic Expansions. Cambridge University Press, London. Cornish, E. A., and R. A. Fisher Ž1937.. ‘‘Moments and cumulants in the specification of distributions.’’ Re®. Internat. Statist. Inst., 5, 307᎐320. Corwin, L. J., and R. H. Szczarba Ž1982.. Multi®ariable Calculus. Marcel Dekker, New York. Courant, R., and F. John Ž1965.. Introduction to Calculus and Analysis, Vol. 1. Wiley, New York. Cramer, ´ H. Ž1946.. Mathematical Methods of Statistics. Princeton University Press, Princeton. Daniels, H. Ž1954.. ‘‘Saddlepoint approximation in statistics.’’ Ann. Math. Statist., 25, 631᎐650. Dasgupta, P. Ž1968.. ‘‘An approximation to the distribution of sample correlation coefficient, when the population is non-normal.’’ Sankhya, Ser. B., 30, 425᎐428. Davis, H. F. Ž1963.. Fourier Series and Orthogonal Functions. Allyn &Bacon, Boston. Davis, P. J. Ž1975.. Interpolation and Approximation. Dover, New York Davis, P. J., and P. Rabinowitz Ž1975.. Methods of Numerical Integration. Academic Press, New York. De Boor, C. Ž1978.. A Practical Guide to Splines. Springer-Verlag, New York. DeBruijn, N. G. Ž1961.. Asymptotic Methods in Analysis, 2nd ed. North-Holland, Amesterdam. DeCani, J. S., and R. A. Stine Ž1986.. ‘‘A note on deriving the information matrix for a logistic distribution.’’ Amer. Statist., 40, 220᎐222. Dempster, A. P., N. M. Laird, and D. B. Rubin Ž1977.. ‘‘Maximum likelihood from incomplete data via the EM algorithm.’’ J. Roy. Statist. Soc. Ser. B, 39, 1᎐38. Divgi, D. R. Ž1979.. ‘‘Calculation of univariate and bivariate normal probability functions.’’ Ann. Statist., 7, 903᎐910.

GENERAL BIBLIOGRAPHY

655

Draper, N. R. Ž1963.. ‘‘Ridge analysis of response surfaces.’’ Technometrics, 5, 469᎐479. Draper, N. R., and A. M. Herzberg Ž1987.. ‘‘A ridge-regression sidelight.’’ Amer. Statist., 41, 282᎐283. Draper, N. R., and H. Smith Ž1981.. Applied Regression Analysis, 2nd ed. Wiley, New York. Draper, N. R., I. Guttman, and P. Lipow Ž1977.. ‘‘All-bias designs for spline functions joined at the axes.’’ J. Amer. Statist. Assoc., 72, 424᎐429. Dugundji, J. Ž1966.. Topology. Allyn and Bacon, Boston. Durbin, J., and G. S. Watson Ž1950.. ‘‘Testing for serial correlation in least squares regression, I.’’ Biometrika, 37, 409᎐428. Durbin, J., and G. S. Watson Ž1951.. ‘‘Testing for serial correlation in least squares regression, II.’’ Biometrika, 38, 159᎐178. Eggermont, P. P. B. Ž1988.. ‘‘Noncentral difference quotients and the derivative.’’ Amer. Math. Monthly, 95, 551᎐553. Eubank, R. L. Ž1984.. ‘‘Approximate regression models and splines.’’ Comm. Statist. Theory Methods, 13, 433᎐484. Evans, M., and T. Swartz Ž1995.. ‘‘Methods for approximating integrals in statistics with special emphasis on Bayesian integration problems.’’ Statist. Sci., 10, 254᎐272. Everitt, B. S. Ž1987.. Introduction to Optimization Methods and Their Application in Statistics. Chapman and Hall, London. Eves, H. Ž1976.. An Introduction to the History of Mathematics, 4th ed. Holt, Rinehart and Winston, New York. Fedorov, V. V. Ž1972.. Theory of Optimal Experiments. Academic Press, New York. Feller, W. Ž1968.. An Introduction to Probability Theory and Its Applications, Vol. I, 3rd ed. Wiley, New York. Fettis, H. E. Ž1976.. ‘‘Fourier series expansions for Pearson Type IV distributions and probabilities.’’ SIAM J. Applied Math., 31, 511᎐518. Fichtali, J., F. R. Van De Voort, and A. I. Khuri Ž1990.. ‘‘Multiresponse optimization of acid casein production.’’ J. Food Process Eng., 12, 247᎐258. Fisher, R. A., and E. Cornish Ž1960.. ‘‘The percentile points of distribution having known cumulants.’’ Technometrics, 2, 209᎐225. Fisher, R. A., A. S. Corbet, and C. B. Williams Ž1943.. ‘‘The relation between the number of species and the number of individuals in a random sample of an animal population.’’ J. Anim. Ecology, 12, 42᎐58. Fisz, M. Ž1963.. Probability Theory and Mathematical Statistics, 3rd ed. Wiley, New York. Fletcher, R. Ž1987.. Practical Methods of Optimization, 2nd ed. Wiley, New York. Fletcher, R., and M. J. D. Powell Ž1963.. ‘‘A rapidly convergent descent method for minimization.’’ Comput. J., 6, 163᎐168. Flournoy, N., and R. K. Tsutakawa, eds. Ž1991.. Statistical Multiple Integration, Contemporary Mathematics 115. Amer. Math. Soc., Providence, Rhode Island. Freud, G. Ž1971.. Orthogonal Polynomials. Pergamon Press, Oxford. Fulks, W. Ž1978.. Ad®anced Calculus, 3rd ed. Wiley, New York. Fuller, W. A. Ž1976.. Introduction to Statistical Time Series. Wiley, New York.

656

GENERAL BIBLIOGRAPHY

Gallant, A. R., and W. A. Fuller Ž1973.. ‘‘Fitting segmented polynomial regression models whose join points have to be estimated.’’ J. Amer. Statist. Assoc., 68, 144᎐147. Gantmacher, F. R. Ž1959.. The Theory of Matrices, Vols. I and II. Chelsea, New York. Georgiev, A. A. Ž1984.. ‘‘Kernel estimates of functions and their derivatives with applications.’’ Statist. Prob. Lett., 2, 45᎐50. Geyer, C. J. Ž1992.. ‘‘Practical Markov chain Monte Carlo.’’ Statist. Sci., 7, 473᎐511. Ghazal, G. A. Ž1994.. ‘‘Moments of the ratio of two dependent quadratic forms.’’ Statist. Prob. Lett., 20, 313᎐319. Gibaldi, M., and D. Perrier Ž1982.. Pharmacokinetics, 2nd ed. Dekker, New York. Gillespie, R. P. Ž1954.. Partial Differentiation. Oliver and Boyd, Edinburgh. Gillespie, R. P. Ž1959.. Integration. Oliver and Boyd, London. Golub, G. H., and C. F. Van Loan Ž1983.. Matrix Computations. Johns Hopkins University Press, Baltimore. Good, I. J. Ž1969.. ‘‘Some applications of the singular decomposition of a matrix.’’ Technometrics, 11, 823᎐831. Goutis, C., and G. Casella Ž1999.. ‘‘Explaining the saddlepoint approximation.’’ Amer. Statist., 53, 216᎐224. Graybill, F. A. Ž1961.. An Introduction to Linear Statistical Models, Vol. I. McGraw-Hill, New York. Graybill, F. A. Ž1976.. Theory and Application of the Linear Model. Duxbury, North Scituate, Massachusetts. Graybill, F. A. Ž1983.. Matrices with Applications in Statistics, 2nd ed. Wadsworth, Belmont, California. Gurland, J. Ž1953.. ‘‘Distributions of quadratic forms and ratios of quadratic forms.’’ Ann. Math. Statist., 24, 416᎐427. Haber, S. Ž1970.. ‘‘Numerical evaluation of multiple integrals.’’ SIAM Re®., 12, 481᎐526. Hall, C. A. Ž1968.. ‘‘On error bounds for spline interpolation.’’ J. Approx. Theory, 1, 209᎐218. Hall, C. A., and W. W. Meyer Ž1976.. ‘‘Optimal error bounds for cubic spline interpolation.’’ J. Approx. Theory, 16, 105᎐122. Hardy, G. H. Ž1955.. A Course of Pure Mathematics, 10th ed. The University Press, Cambridge, England. Hardy, G. H., J. E. Littlewood, and G. Polya ´ Ž1952.. Inequalities, 2nd ed. Cambridge University Press, Cambridge, England. Harris, B. Ž1966.. Theory of Probability. Addison-Wesley, Reading, Massachusetts. Hartig, D. Ž1991.. ‘‘L’Hopital’s rule via integration.’’ Amer. Math. Monthly, 98, ˆ 156᎐157. Hartley, H. O., and J. N. K. Rao Ž1967.. ‘‘Maximum likelihood estimation for the mixed analysis of variance model.’’ Biometrika, 54, 93᎐108. Healy, M. J. R. Ž1986.. Matrices for Statistics. Clarendon Press, Oxford, England.

GENERAL BIBLIOGRAPHY

657

Heiberger, R.M., P. F. Velleman, and M. A. Ypelaar Ž1983.. ‘‘Generating test data with independently controllable features for multivariate general linear forms.’’ J. Amer. Statist. Assoc., 78, 585᎐595. Henderson, H. V., and S. R. Searle Ž1981.. ‘‘The vec-permutation matrix, the vec operator and Kronecker products: A review.’’ Linear and Multilinear Algebra, 9, 271᎐288. Henderson, H. V., and F. Pukelsheim, and S. R. Searle Ž1983.. ‘‘On the history of the Kronecker product.’’ Linear and Multilinear Algebra, 14, 113᎐120. Henle, J. M., and E. M. Kleinberg Ž1979.. Infinitesimal Calculus. The MIT Press, Cambridge, Massachusetts. Hillier, F. S., and G. J. Lieberman Ž1967.. Introduction to Operations Research. Holden-Day, San Francisco. Hirschman, I. I., Jr. Ž1962.. Infinite Series. Holt, Rinehart and Winston, New York. Hoerl, A. E. Ž1959.. ‘‘Optimum solution of many variables equations.’’ Chem. Eng. Prog., 55, 69᎐78. Hoerl, A. E., and R. W. Kennard Ž1970a.. ‘‘Ridge regression: Biased estimation for non-orthogonal problems.’’ Technometrics, 12, 55᎐67. Hoerl, A. E., and R. W. Kennard Ž1970b.. ‘‘Ridge regression: Applications to non-orthogonal problems.’’ Technometrics, 12, 69᎐82; correction, 12, 723. Hogg, R. V., and A. T. Craig Ž1965.. Introduction to Mathematical Statistics, 2nd ed. Macmillan, New York. Huber, P. J. Ž1973.. ‘‘Robust regression: Asymptotics, conjectures and Monte Carlo.’’ Ann. Statist. 1, 799᎐821. Huber, P. J. Ž1981.. Robust Statistics. Wiley, New York. Hyslop, J. M. Ž1954.. Infinite Series, 5th ed. Oliver and Boyd, Edinburgh, England. Jackson, D. Ž1941.. Fourier Series and Orthogonal Polynomials. Mathematical Association of America, Washington. James, A. T. Ž1954.. ‘‘Normal multivariate analysis and the orthogonal group.’’ Ann. Math. Statist., 25, 40᎐75. James, A. T., and R. A. J. Conyers Ž1985.. ‘‘Estimation of a derivative by a difference quotient: Its application to hepatocyte lactate metabolism.’’ Biometrics, 41, 467᎐476. Johnson, M. E., and C. J. Nachtsheim Ž1983.. ‘‘Some guidelines for constructing exact D-optimal designs on convex design spaces.’’ Technometrics, 25, 271᎐277. Johnson, N. L., and S. Kotz Ž1968.. ‘‘Tables of distributions of positive definite quadratic forms in central normal variables.’’ Sankhya, Ser. B, 30, 303᎐314. Johnson, N. L., and S. Kotz Ž1969.. Discrete Distributions. Houghton Mifflin, Boston. Johnson, N. L., and S. Kotz Ž1970a.. Continuous Uni®ariate Distributionsᎏ1. Houghton Mifflin, Boston. Johnson, N. L., and S. Kotz Ž1970b.. Continuous Uni®ariate Distributionsᎏ2. Houghton Mifflin, Boston. Johnson, P. E. Ž1972.. A History of Set Theory. Prindle, Weber, and Schmidt, Boston. Jones, E. R., and T. J. Mitchell Ž1978.. ‘‘Design criteria for detecting model inadequacy.’’ Biometrika, 65, 541᎐551.

658

GENERAL BIBLIOGRAPHY

Judge, G. G., W. E. Griffiths, R. C. Hill, and T. C. Lee Ž1980.. The Theory and Practice of Econometrics. Wiley, New York. Kahaner, D. K. 1991. ‘‘A survey of existing multidimensional quadrature routines.’’ In Statistical Multiple Integration, Contemporary Mathematics 115, N. Flournoy and R. K. Tsutakawa, eds. Amer. Math. Soc., Providence, pp. 9᎐22. Kaplan, W. Ž1991.. Ad®anced Calculus, 4th ed. Addison-Wesley, Redwood City, California. Kaplan, W., and D. J. Lewis Ž1971.. Calculus and Linear Algebra, Vol. II. Wiley, New York. Karson, M. J., A. R. Manson, and R. J. Hader Ž1969.. ‘‘Minimum bias estimation and experimental design for response surfaces.’’ Technometrics, 11, 461᎐475. Kass, R. E., L. Tierney, and J. B. Kadane Ž1991.. ‘‘Laplace’s method in Bayesian analysis.’’ In Statistical Multiple Integration, Contemporary Mathematics 115, N. Flournoy and R. K. Tsutakawa, eds. Amer. Math. Soc., Providence, pp. 89᎐99. Katz, D., and D. Z. D’Argenio Ž1983.. ‘‘Experimental design for estimating integrals by numerical quadrature, with applications to pharmacokinetic studies.’’ Biometrics, 39, 621᎐628. Kawata, T. Ž1972.. Fourier Analysis in Probability Theory. Academic Press, New York. Kendall, M., and A. Stuart Ž1977.. The Ad®anced Theory of Statistics, Vol. 1, 4th ed. Macmillian, New York. Kerridge, D. F., and G. W. Cook Ž1976.. ‘‘Yet another series for the normal integral.’’ Biometrika, 63, 401᎐403. Khuri, A. I. Ž1982.. ‘‘Direct products: A powerful tool for the analysis of balanced data.’’ Comm. Statist. Theory Methods, 11, 2903᎐2920. Khuri, A. I. Ž1984.. ‘‘Interval estimation of fixed effects and of functions of variance components in balanced mixed models.’’ Sankhya, Ser. B, 46, 10᎐28. Khuri, A. I., and G. Casella Ž2002.. ‘‘The existence of the first negative moment revisited.’’ Amer. Statist., 56, 44᎐47. Khuri, A. I., and M. Conlon Ž1981.. ‘‘Simultaneous optimization of multiple responses represented by polynomial regression functions.’’ Technometrics, 23, 363᎐375. Khuri, A. I., and J. A. Cornell Ž1996.. Response Surfaces, 2nd ed. Marcel Dekker, New York. Khuri, A. I., and I. J. Good Ž1989.. ‘‘The parameterization of orthogonal matrices: A review mainly for statisticians.’’ South African Statist. J., 23, 231᎐250. Khuri, A. I., and R. H. Myers Ž1979.. ‘‘Modified ridge analysis.’’ Technometrics, 21, 467᎐473. Khuri, A. I., and R. H. Myers Ž1981.. ‘‘Design related robustness of tests in regression models.’’ Comm. Statist. Theory Methods, 10, 223᎐235. Khuri, A. I., and H. Sahai Ž1985.. ‘‘Variance components analysis: A selective literature survey.’’ Internat. Statist. Re®, 53, 279᎐300. Kiefer, J. Ž1958.. ‘‘On the nonrandomized optimality and the randomized nonoptimality of symmetrical designs.’’ Ann. Math. Statist., 29, 675᎐699. Kiefer, J. Ž1959.. ‘‘Optimum experimental designs’’ Žwith discussion .. J. Roy. Statist. Soc., Ser. B, 21, 272᎐319.

GENERAL BIBLIOGRAPHY

659

Kiefer, J. Ž1960.. ‘‘Optimum experimental designs V, with applications to systematic and rotatable designs.’’ In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1. University of California Press, Berkeley, pp. 381᎐405. Kiefer, J. Ž1961.. ‘‘Optimum designs in regression problems II.’’ Ann. Math. Statist., 32, 298᎐325. Kiefer, J. Ž1962a.. ‘‘Two more criteria equivalent to D-optimality of designs.’’ Ann. Math. Statist., 33, 792᎐796. Kiefer, J. Ž1962b.. ‘‘An extremum result.’’ Canad. J. Math., 14, 597᎐601. Kiefer, J. Ž1975.. ‘‘Optimal design: Variation in structure and performance under change of criterion.’’ Biometrika, 62, 277᎐288. Kiefer, J., and J. Wolfowitz Ž1960.. ‘‘The equivalence of two extremum problems.’’ Canad. J. Math., 12, 363᎐366. Kirkpatrick, S., C. D. Gelatt, and M. P. Vechhi Ž1983.. ‘‘Optimization by simulated annealing,’’ Science, 220, 671᎐680. Knopp, K. Ž1951.. Theory and Application of Infinite Series. Blackie and Son, London. Kosambi, D. D. Ž1949.. ‘‘Characteristic properties of series distributions.’’ Proc. Nat. Inst. Sci. India, 15, 109᎐113. Krylov, V. I. Ž1962.. Approximate Calculation of Integrals. Macmillan, New York. Kufner, A., and J. Kadlec Ž1971.. Fourier Series. Iliffe BooksᎏThe Butterworth Group, London. Kupper, L. L. Ž1972.. ‘‘Fourier series and spherical harmonics regression.’’ Appl. Statist., 21, 121᎐130. Kupper, L. L. Ž1973.. ‘‘Minimax designs for Fourier series and spherical harmonics regressions: A characterization of rotatable arrangements.’’ J. Roy. Statist. Soc., Ser. B, 35, 493᎐500. Lancaster, P. Ž1969.. Theory of Matrices. Academic Press, New York. Lancaster, P., and K. Salkauskas Ž1986.. Cur®e and Surface Fitting. Academic Press, London. Lange, K. Ž1999.. Numerical Analysis for Statisticians. Springer, New York. Lehmann, E. L. Ž1983.. Theory of Point Estimation. Wiley, New York. Lieberman, G. J., and Owen, D. B. Ž1961.. Tables of the Hypergeometric Probability Distribution. Stanford University Press, Palo Alto, California. Lieberman, O. Ž1994.. ‘‘A Laplace approximation to the moments of a ratio of quadratic forms.’’ Biometrika, 81, 681᎐690. Lindgren, B. W. Ž1976.. Statistical Theory, 3rd ed. Macmillan, New York. Little, R. J. A., and D. B. Rubin Ž1987.. Statistical Analysis with Missing Data. Wiley, New York. Liu, Q., and D. A. Pierce Ž1994.. ‘‘A note on Gauss᎐Hermite quadrature.’’ Biometrika, 81, 624᎐629. Lloyd, E. Ž1980.. Handbook of Applicable Mathematics, Vol. II. Wiley, New York. Lowerre, J. M. Ž1982.. ‘‘An introduction to modern matrix methods and statistics.’’ Amer. Statist., 36, 113᎐115.

660

GENERAL BIBLIOGRAPHY

Lowerre, J. M. Ž1983.. ‘‘An integral of the bivariate normal and an application.’’ Amer. Statist., 37, 235᎐236. Lucas, J. M. Ž1976.. ‘‘Which response surface design is best.’’ Technometrics, 18, 411᎐417. Luceno, ˜ A. Ž1997.. ‘‘Further evidence supporting the numerical usefulness of characteristic functions.’’ Amer. Statist., 51, 233᎐234. Lukacs, E. Ž1970.. Characteristic Functions, 2nd ed. Hafner, New York. Magnus, J. R., and H. Neudecker Ž1988.. Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York. Mandel, J. Ž1982.. ‘‘Use of the singular-value decomposition in regression analysis.’’ Amer. Statist., 36, 15᎐24. Marcus, M., and H. Minc Ž1988.. Introduction to Linear Algebra. Dover, New York. Mardia, K. V., and T. W. Sutton Ž1978.. ‘‘Model for cylindrical variables with applications.’’ J. Roy. Statist. Soc., Ser. B, 40, 229᎐233. Marsaglia, G., and G. P. H. Styan Ž1974.. ‘‘Equalities and inequalities for ranks of matrices.’’ Linear and Multilinear Algebra, 2, 269᎐292. May, W. G. Ž1970.. Linear Algebra. Scott, Foresman and Company, Glenview, Illinois. McCullagh, P. Ž1994.. ‘‘Does the moment-generating function characterize a distribution?’’ Amer. Statist., 48, 208. Menon, V. V., B. Prasad, and R. S. Singh Ž1984.. ‘‘Non-parametric recursive estimates of a probability density function and its derivatives.’’ J. Statist. Plann. Inference, 9, 73᎐82. Miller, R. G., Jr. Ž1981.. Simultaneous Statistical Inference, 2nd ed. Springer, New York. Milliken, G. A., and D. E. Johnson Ž1984.. Analysis of Messy Data. Lifetime Learning Publications, Belmont, California. Mitchell, T. J. Ž1974.. ‘‘An algorithm for the construction of D-optimal experimental designs.’’ Technometrics, 16, 203᎐210. Montgomery, D. C., and E. A. Peck Ž1982.. Introduction to Linear Regression Analysis. Wiley, New York. Moran, P. A. P. Ž1968.. An Introduction to Probability Theory. Clarendon Press, Oxford. Morin, D. Ž1992.. ‘‘Exact moments of ratios of quadratic forms.’’ Metron, 50, 59᎐78. Morland, T. Ž1998.. ‘‘Approximations to the normal distribution function.’’ Math. Gazette, 82, 431᎐437. Morrison, D. F. Ž1967.. Multi®ariate Statistical Methods. McGraw-Hill, New York. Muirhead, R. J. Ž1982.. Aspects of Multi®ariate Statistical Theory. Wiley, New York. Myers, R. H. Ž1976.. Response Surface Methodology. Author, Blacksburg, Virginia. Myers, R. H. Ž1990.. Classical and Modern Regression with Applications, 2nd ed. PWS-Kent, Boston. Myers, R. H., and W. H. Carter, Jr. Ž1973.. ‘‘Response surface techniques for dual response systems.’’ Technometrics, 15, 301᎐317. Myers, R. H., and A. I. Khuri Ž1979.. ‘‘A new procedure for steepest ascent.’’ Comm. Statist. Theory Methods, 8, 1359᎐1376.

GENERAL BIBLIOGRAPHY

661

Myers, R. H., A. I. Khuri, and W. H. Carter, Jr. Ž1989.. ‘‘Response surface methodology: 1966᎐1988.’’ Technometrics, 31, 137᎐157. Nelder, J. A., and R. Mead Ž1965.. ‘‘A simplex method for function minimization.’’ Comput. J., 7, 308᎐313. Nelson, L. S. Ž1973.. ‘‘A sequential simplex procedure for non-linear least-squares estimation and other function minimization problems.’’ In 27th Annual Technical Conference Transactions, American Society for Quality Control, pp. 107᎐117. Newcomb, R. W. Ž1960.. ‘‘On the simultaneous diagonalization of two semidefinite matrices.’’ Quart. Appl. Math., 19, 144᎐146. Nonweiler, T. R. F. Ž1984.. Computational Mathematics. Ellis Horwood, Chichester, England. Nurcombe, J. R, Ž1979.. ‘‘A sequence of convergence tests.’’ Amer. Math. Monthly, 86, 679᎐681. Ofir, C., and A. I. Khuri Ž1986.. ‘‘Multicollinearity in marketing models: Diagnostics and remedial measures.’’ Internat. J. Res. Market., 3, 181᎐205. Olkin, I. Ž1990.. ‘‘Interface between statistics and linear algebra.’’ In Matrix Theory and Applications, Vol. 40, C. R. Johnson, ed., American Mathematical Society, Providence, pp. 233᎐256. Olsson, D. M., and L. S. Nelson Ž1975.. ‘‘The Nelder᎐Mead simplex procedure for function minimization.’’ Technometrics, 17, 45᎐51. Otnes, R. K., and L. Enochson Ž1978.. Applied Time Series Analysis. Wiley, New York. Park, S. H. Ž1978.. ‘‘Experimental designs for fitting segmented polynomial regression models.’’ Technometrics, 20, 151᎐154. Parzen, E. Ž1962.. Stochastic Processes. Holden-Day, San Francisco. Pazman, A. Ž1986.. Foundations of Optimum Experimental Design. Reidel, Dordrecht, Holland. Perez-Abreu, V. Ž1991.. ‘‘Poisson approximation to power series distributions.’’ Amer. ´ Statist., 45, 42᎐45. Pfeiffer, P. E. Ž1990.. Probability for Applications. Springer, New York. Phillips, C., and B. Cornelius Ž1986.. Computational Numerical Methods. Ellis Horwood, Chichester, England. Phillips, G. M., and P. J. Taylor Ž1973.. Theory and Applications of Numerical Analysis. Academic Press, New York. Piegorsch, W. W., and A. J. Bailer Ž1993.. ‘‘Minimum mean-square error quadrature.’’ J. Statist. Comput. Simul., 46, 217᎐234. Piegorsch, W. W., and G. Casella Ž1985.. ‘‘The existence of the first negative moment.’’ Amer. Statist., 39, 60᎐62. Pinkus, A., and S. Zafrany Ž1997.. Fourier Series and Integral Transforms. Cambridge University Press, Cambridge, England. Plackett, R. L., and J. P. Burman Ž1946.. ‘‘The design of optimum multifactorial experiments.’’ Biometrika, 33, 305᎐325. Poirier, D. J. Ž1973.. ‘‘Piecewise regression using cubic splines.’’ J. Amer. Statist. Assoc., 68, 515᎐524. Powell, M. J. D. Ž1967.. ‘‘On the maximum errors of polynomial approximations defined by interpolation and by least squares criteria.’’ Comput. J., 9, 404᎐407.

662

GENERAL BIBLIOGRAPHY

Prenter, P. M. Ž1975.. Splines and Variational Methods. Wiley, New York. Price, G. B. Ž1947.. ‘‘Some identities in the theory of determinants.’’ Amer. Math. Monthly, 54, 75᎐90. Price, W. L. Ž1977.. ‘‘A controlled random search procedure for global optimization.’’ Comput. J., 20, 367᎐370. Pye, W. C., and P. G. Webster Ž1989.. ‘‘A note on Raabe’s test extended.’’ Math. Comput. Ed., 23, 125᎐128. Ralston, A., and P. Rabinowitz Ž1978.. A First Course in Numerical Analysis. McGraw-Hill, New York. Ramsay, J. O. Ž1988.. ‘‘Monotone regression splines in action.’’ Statist. Sci., 3, 425᎐461. Randles, R. H., and D. A. Wolfe Ž1979.. Introduction to the Theory of Nonparametric Statistics. Wiley, New York. Rao, C. R. Ž1970.. ‘‘Estimation of heteroscedastic variances in linear models.’’ J. Amer. Statist. Assoc., 65, 161᎐172. Rao, C. R. Ž1971.. ‘‘Estimation of variance and covariance componentsᎏMINQUE theory.’’ J. Multi®ariate Anal., 1, 257᎐275. Rao, C. R. Ž1972.. ‘‘Estimation of variance and covariance components in linear models.’’ J. Amer. Statist. Assoc., 67, 112᎐115. Rao, C. R. Ž1973.. Linear Statistical Inference and its Applications, 2nd ed. Wiley, New York. Reid, W. H., and S. J. Skates Ž1986.. ‘‘On the asymptotic approximation of integrals.’’ SIAM J. Appl. Math, 46, 351᎐358. Rice, J. R. Ž1969.. The Approximation of Functions, Vol. 2. Addison-Wesley, Reading, Massachusetts. Rivlin, T. J. Ž1969.. An Introduction to the Approximation of Functions. Dover, New York. Rivlin, T. J. Ž1990.. Chebyshe® Polynomials, 2nd ed. Wiley, New York. Roberts, A. W., and D. E. Varberg Ž1973.. Con®ex Functions. Academic Press, New York. Rogers, G. S. Ž1984.. ‘‘Kronecker products in ANOVAᎏA first step.’’ Amer. Statist., 38, 197᎐202. Roussas, G. G. Ž1973.. A First Course in Mathematical Statistics. Addison-Wesley, Reading, Massachusetts. Rudin, W. Ž1964.. Principles of Mathematical Analysis, 2nd ed. McGraw-Hill, New York. Rustagi, J. S., ed. Ž1979.. Optimizing Methods in Statistics. Academic Press, New York. Sagan, H. Ž1974.. Ad®anced Calculus. Houghton Mifflin, Boston. Satterthwaite, F. E. Ž1946.. ‘‘An approximate distribution of estimates of variance components.’’ Biometrics Bull., 2, 110᎐114. Scheffe, ´ H. Ž1959.. The Analysis of Variance. Wiley, New York. Schoenberg, I. J. Ž1946.. ‘‘Contributions to the problem of approximation of equidistant data by analytic functions.’’ Quart. Appl. Math., 4, 45᎐99; 112᎐141.

GENERAL BIBLIOGRAPHY

663

Schone, ¨ A., and W. Schmid Ž2000.. ‘‘On the joint distribution of a quadratic and a linear form in normal variables.’’ J. Mult. Anal., 72, 163᎐182. Schwartz, S. C. Ž1967.. ‘‘Estimation of probability density by an orthogonal series’’, Ann. Math. Statist., 38, 1261᎐1265. Searle, S. R. Ž1971.. Linear Models. Wiley, New York. Searle, S. R. Ž1982.. Matrix Algebra Useful for Statistics. Wiley, New York. Seber, G. A. F. Ž1984.. Multi®ariate Obser®ations. Wiley, New York. Shoup, T. E. Ž1984.. Applied Numerical Methods for the Microcomputer. Prentice-Hall, Englewood Cliffs, New Jersey. Silvey, S. D. Ž1980.. Optimal Designs. Chapman and Hall, London. Smith, D. E. Ž1958.. History of Mathematics, Vol. I. Dover, New York. Smith, P. L. Ž1979.. ‘‘Splines as a useful and convenient statistical tool.’’ Amer. Statist., 33, 57᎐62. Spendley, W., G. R. Hext, and F. R. Himsworth Ž1962.. ‘‘Sequential application of simplex designs in optimization and evolutionary operation.’’ Technometrics, 4, 441᎐461. St. John, R. C., and N. R. Draper Ž1975.. ‘‘ D-optimality for regression designs: A review.’’ Technometrics, 17, 15᎐23. Stark, P. A. Ž1970.. Introduction to Numerical Methods. Macmillan, London. Stoll, R. R. Ž1963.. Set Theory and Logic. W. H. Freeman, San Francisco. Strawderman, R. L. Ž2000.. ‘‘Higher-order asymptotic approximation: Laplace, saddlepoint, and related methods.’’ J. Amer. Statist. Assoc., 95, 1358᎐1364. Stroud, A. H. Ž1971.. Approximate Calculation of Multiple Integrals. Prentice-Hall, Englewood Cliffs, New Jersey. Subrahmaniam, K. Ž1966.. ‘‘Some contributions to the theory of non-normalityᎏI Žunivariate case..’’ Sankhya, Ser. A, 28, 389᎐406. Sutradhar, B. C., and R. F. Bartlett Ž1989.. ‘‘An approximation to the distribution of the ratio of two general quadratic forms with application to time series valued designs.’’ Comm. Statist. Theory Methods, 18, 1563᎐1588. Swallow, W. H., and S. R. Searle Ž1978.. ‘‘Minimum variance quadratic unbiased estimation ŽMIVQUE. of variance components.’’ Technometrics, 20, 265᎐272. Szego, ¨ G. Ž1975.. Orthogonal Polynomials, 4th ed. Amer. Math. Soc., Providence, Rhode Island. Szidarovszky, F., and S. Yakowitz Ž1978.. Principles and Procedures of Numerical Analysis. Plenum Press, New York. Taylor, A. E., and W. R. Mann Ž1972.. Ad®anced Calculus, 2nd ed. Wiley, New York. Thibaudeau, Y., and G. P. H. Styan Ž1985.. ‘‘Bounds for Chakrabarti’s measure of imbalance in experimental design.’’ In Proceedings of the First International Tampere Seminar on Linear Statistical Models and Their Applications, T. Pukkila and S. Puntanen, eds. University of Tampere, Tampere, Finland, pp. 323᎐347. Thiele, T. N. Ž1903.. Theory of Obser®ations. Layton, London. Reprinted in Ann. Math. Statist. Ž1931., 2, 165᎐307.

664

GENERAL BIBLIOGRAPHY

Tierney, L., R. E. Kass, and J. B. Kadane Ž1989.. ‘‘Fully exponential Laplace approximations to expectations and variances of nonpositive functions.’’ J. Amer. Statist. Assoc., 84, 710᎐716. Tiku, M. L. Ž1964a.. ‘‘Approximating the general non-normal variance ratio sampling distributions.’’ Biometrika, 51, 83᎐95. Tiku, M. L. Ž1964b.. ‘‘A note on the negative moments of a truncated Poisson variate.’’ J. Amer. Statist. Assoc., 59, 1220᎐1224. Tolstov, G. P. Ž1962.. Fourier Series. Dover, New York. ŽTranslated from the Russian by Richard A. Silverman.. Tucker, H. G., Ž1962.. Probability and Mathematical Statistics. Academic Press, New York. Vilenkin, N. Y. Ž1968.. Stories about Sets. Academic Press, New York. Viskov, O. V. Ž1992.. ‘‘Some remarks on Hermite polynomials.’’ Theory Prob. Appl., 36, 633᎐637. Waller, L. A. Ž1995.. ‘‘Does the characteristic function numerically distinguish distributions?’’ Amer. Statist., 49, 150᎐152. Waller, L. A., B. W. Turnbull, and J. M. Hardin Ž1995.. ‘‘Obtaining distribution functions by numerical inversion of characteristic functions with applications.’’ Amer. Statist., 49, 346᎐350. Watson, G. S. Ž1964.. ‘‘A note on maximum likelihood.’’ Sankhya, Ser. A, 26, 303᎐304. Weaver, H. J. Ž1989.. Theory of Discrete and Continuous Fourier Analysis. Wiley, New York. Wegman, E. J., and I. W. Wright Ž1983.. ‘‘Splines in statistics.’’ J. Amer. Statist. Assoc., 78, 351᎐365. Wen, L. Ž2001.. ‘‘A counterexample for the two-dimensional density function.’’ Amer. Math. Monthly, 108, 367᎐368. Wetherill, G. B., P. Duncombe, M. Kenward, J. Kollerstrom, ¨ ¨ S. R. Paul, and B. J. Vowden Ž1986.. Regression Analysis with Applications. Chapman and Hall, London. Wilks, S. S. Ž1962.. Mathematical Statistics. Wiley, New York. Withers, C. S. Ž1984.. ‘‘Asymptotic expansions for distributions and quantiles with power series cumulants.’’ J. Roy. Statist. Soc., Ser. B, 46, 389᎐396. Wold, S. Ž1974.. ‘‘Spline functions in data analysis.’’ Technometrics, 16, 1᎐11. Wolkowicz, H., and G. P. H. Styan Ž1980.. ‘‘Bounds for eigenvalues using traces.’’ Linear Algebra Appl., 29, 471᎐506. Wong, R. Ž1989.. Asymptotic Approximations of Integrals. Academic Press, New York. Woods, J. D., and H. O. Posten Ž1977.. ‘‘The use of Fourier series in the evaluation of probability distribution functions.’’ Comm. Statist.ᎏSimul. Comput., 6, 201᎐219. Wynn, H. P. Ž1970.. ‘‘The sequential generation of D-optimum experimental designs.’’ Ann. Math. Statist., 41, 1655᎐1664. Wynn, H. P. Ž1972.. ‘‘Results in the theory and construction of D-optimum experimental designs.’’ J. Roy. Statist. Soc., Ser. B, 34, 133᎐147. Zanakis, S. H., and J. S. Rustagi, eds. Ž1982.. Optimization in Statistics. North-Holland, Amsterdam. Zaring, W. M. Ž1967.. An Introduction to Analysis. Macmillian, New York.

Index

Absolute error loss, 86 Absolutely continuous function, 91 random variable, 116, 239 Addition of matrices, 29 Adjoint, 34 Adjugate, 34 Admissible estimator, 86 Approximation of density functions, 456 functions, 403, 495 integrals, 517, 533 normal integral, 460, 469 quantiles of distributions, 456, 458, 468 Arcsin x, 6, 78 Asymptotically equal, 66, 147 Asymptotic distributions, 120 Average squared bias, 360, 431 Average variance, 360, 400 Balanced mixed model, 43 Behrens᎐Fisher test, 324 Bernoulli sequence, 14 Bernstein polynomial, 406, 410 Best linear unbiased estimator, 312 Beta function, 256 Bias, 359, 360 criterion, 359, 366 in the model, 359, 366 Biased estimator, 196 Binomial distribution, 194 Binomial random variable, 14, 203 Bolzano᎐Weierstrass theorem, 11 Borel field, 13 Boundary, 262, 297 point, 262 Bounded sequence, 132, 262 Bounded set, 9, 13, 72, 262

Bounded variation, function of, 210, 212 Bounds on eigenvalues, 41 Box and Cox transformation, 120 Box᎐Draper determinant criterion, 400 Box᎐Lucas criterion, 368, 422, 424᎐425 Calibration, 242 Canonical correlation coefficient, 56 Cartesian product, 3, 21 Cauchy criterion, 137, 221 Cauchy distribution, 240 Cauchy᎐Schwarz inequality, 25, 229 Cauchy sequence, 138, 222 Cauchy’s condensation test, 153 Cauchy’s mean value theorem, 103, 105, 129 Cauchy’s product, 163 Cauchy’s test, 149 Cell, 294 Center-point replications, 400 Central limit theorem, 120, 516 Chain rule, 97, 116 Change of variables, 219, 299 Characteristic equation, 37 Characteristic function, 505 inversion formula for, 507 Characteristic root, 36 Characteristic vector, 37 Chebyshev polynomials, 415, 444 discrete, 462 of the first kind, 444 of the second kind, 445 zeros of, 415, 444 Chebyshev’s inequality, 19, 184, 251, 259 Chi-squared random variable, 246 Christoffel’s identity, 443 Closed set, 11᎐13, 72, 262 Coded levels, 347 Coefficient of variation, 242

665

666 Cofactor, 31, 357 Compact set, 13 Comparison test, 144 Complement of a set, 2 relative, 2 Composite function, 6, 97, 265, 276 Concave function, 79 Condition number, 47 Confidence intervals, simultaneous, 382, 384, 388 Consistent estimator, 83 Constrained maximization, 329 Constrained optimization, 288 Constraints, equality, 288 Continuous distribution, 14, 82 Continuous function, 66, 210, 264 absolutely, 91 left, 68 piecewise, 481, 486, 489, 498 right, 68, 83 Continuous random variable, 14, 82, 116, 239 Contrast, 384 Controlled random search procedure, 336, 426 Convergence almost surely, 192 in distribution, 120 in probability, 83, 191 in quadratic mean, 191 Convex function, 79, 84, 98 Convex set, 79 Convolution, 499 Cornish᎐Fisher expansion, 460, 468 Correlation coefficient, 311 Countable set, 6 Covering of a set, 12 open, 12 Cubic spline regression model, 430 Cumulant generating function, 189 Cumulants, 189, 459 Cumulative distribution function, 14, 82, 304 joint, 14 Curvature, in response surface, 342 Cylindrical data, 505 d’Alembert’s test, 148 Davidon᎐Fletcher ᎐Powell method, 331 Deleted neighborhood, 57, 262 Density function, 14, 116, 239, 304 approximation of, 456 bivariate, 305 marginal, 304 Derivative, 93 directional, 273

INDEX

partial, 267 total, 270 Design A-optimal, 365 approximate, 363 Box᎐Behnken, 359 center, 344, 356 central composite, 347, 353, 358, 400 dependence of, 368᎐369, 422 D-optimal, 363, 365, 369 efficiency of, 365 E-optimal, 365 exact, 363 first-order, 340, 356, 358 G-optimal, 364 ⌳ 1-optimal, 399 matrix, 356 measure, 362᎐363, 365 moments, 362 nonlinear model, 367 optimal, 362, 425 orthogonal, 358 parameter-free, 426 Plackett᎐Burman, 358 points, 353, 360 response surface, 339, 355, 359 rotatable, 350, 356 second-order, 344, 358 3 k factorial, 358 2 k factorial, 340, 358 Design measure continuous, 362 discrete, 363, 366 Determinant, 31 Hessian, 276 Jacobian, 268 Vandermonde’s, 411 DETMAX algorithm, 366 Diagonalization of a matrix, 38 Differentiable function, 94, 113 Differential operator, 277 Differentiation under the integral sign, 301 Directional derivative, 273 Direct product, 30, 44 Direct search methods, 332 Direct sum of matrices, 30 of vector subspaces, 25 Discontinuity, 67 first kind, 67 second kind, 67 Discrete distribution, 14 Discrete random variable, 14, 182

667

INDEX

Disjoint sets, 2 Distribution binomial, 194 Cauchy, 240 chi-squared, 245 continuous, 14, 82 discrete, 14 exponential, 20, 91, 131 F, 383᎐384 gamma, 251 hypergeometric, 463 logarithmic series, 193 logistic, 241 marginal, 304 multinomial, 325 negative binomial, 182, 195 normal, 120, 306, 311, 400 Pearson Type IV, 503 Poisson, 119, 183, 204 power series, 193 t, 259 truncated Poisson, 373 uniform, 83, 92, 116, 250 Domain of a function, 5 Dot product, 23 Eigenvalues, 36᎐37 Eigenvectors, 37 EM algorithm, 372, 375 Empty set, 1 Equal in distribution, 15 Equality of matrices, 28 Equivalence class, 4 Equivalence theorem, 365 Equivalent sets, 5 Error loss absolute, 86 squared, 86 Estimable linear function, 45, 382, 387 Estimation of unknown densities, 461 Estimator admissible, 86 best linear unbiased, 312 biased, 196 consistent, 83 least absolute value, 327 M, 327 minimax, 86 minimum norm quadratic unbiased, 380 minimum variance quadratic unbiased, 382 quadratic, 379 ridge, 56, 196 Euclidean norm, 24, 42, 179, 261

Euclidean space, 21 Euler’s theorem, 317 Event, 13 Exchangeable random variables, 15 Exponential distribution, 20, 91, 131 Extrema absolute, 113 local, 113 Factorial moment, 191 Failure rate, 130 F distribution, 383᎐384 Fisher’s method of scoring, 374 Fixed effects, 44 Fourier coefficients, 472 integral, 488 transform, 497, 507 Fourier series, 471 convergence of, 475 differentiation of, 483 integration of, 483 Fubini’s theorem, 297 Function, 5 absolutely continuous, 91 bounded, 65, 72 bounded variation, 210, 212 composite, 6, 97, 265, 276 concave, 79 continuous, 67, 210, 264 convex, 79, 84, 98 differentiable, 94, 113 homogeneous, 317 implicit, 273 inverse, 6, 76, 102 left continuous, 68 limit of, 57 Lipschitz continuous, 75, 409 loss, 86 lower semicontinuous, 90 monotone, 76, 101, 116, 210 multivariable, 261 one-to-one, 5 onto, 5 periodic, 473, 489, 504 Riemann integrable, 206, 210, 215 Riemann᎐Stieltjes integrable, 234 right continuous, 68, 83 risk, 86 uniformly continuous, 68, 74, 82, 265, 407 upper semicontinuous, 90 Functionally dependent, 318

668 Function of a random variable, 116 Fundamental theorem of calculus, 219 Gamma distribution, 251 Gamma function, 246, 532 approximation for, 532 Gaussian quadrature, 524 Gauss᎐Markov theorem, 55 Gauss’s test, 156 Generalized distance approach, 370 Generalized inverse, 36 Generalized simulated annealing method, 338 Generalized variance, 363 Geometric mean, 84, 230 Geometric series, 142 Gibbs phenomenon, 514 Gradient, 275 methods, 329 Gram᎐Charlier series, 457 Gram᎐Schmidt orthonormalization, 24 Greatest-integer function, 238 Greatest lower bound, 9, 72 Harmonic frequencies, 501 mean, 85 series, 145 Hazard rate, 131 Heine᎐Borel theorem, 13 Hermite polynomials, 447 applications of, 456, 542 normalized, 461 Hessian determinant, 276 matrix, 275, 285, 374 Heteroscedasticity, 118 Holder’s inequality, 230᎐231 ¨ Homogeneous function, 317 Euler’s theorem for, 317 Homoscedasticity, 118 Hotelling’s T 2 statistic, 49 Householder matrix, 38 Hypergeometric distribution, 463 probability, 462 series, 157 Idempotent matrix, 38 Ill conditioning, 46 Image of a set, 5 Implicit function, 273 Implicit function theorem, 282 Importance sampling, 537

INDEX

Improper integral, 220 absolutely convergent, 222, 226 conditionally convergent, 222, 226 of the first kind, 220, 528 of the second kind, 221, 225 Indefinite integral, 217 Indeterminate form, 107 Inequality Cauchy᎐Schwarz, 25, 229 Chebyshev’s, 19, 184, 251 Holder’s, 230, 231 ¨ Jensen’s, 84, 233 Markov’s, 19, 185 Minkowski’s, 232 Infimum, 9, 73 Inner product, 23 Input variables, 339, 351 Integral improper, 220 indefinite, 217 Riemann, 206, 293 Riemann᎐Stieltjes, 234 Integral test, 153, 224 Interaction, 43 Interior point, 262 Intermediate-value theorem, 71, 217 Interpolation, 410 error, 414, 416, 423 Lagrange, 411, 419, 422 points, 411᎐412, 414, 423 Intersection, 2 Interval of convergence, 174, 190 Inverse function, 6, 76, 102 Inverse function theorem, 280, 305 Inverse of a matrix, 34 Inverse regression, 242 Irrational number, 9 Jacobian determinant, 268 matrix, 268 Jacobi polynomials, 443 applications of, 462 Jensen’s inequality, 84, 233 Jordan content, 298 Jordan measurable, 298 Kernel of a linear transformation, 26 Khinchine’s theorem, 192 Knots, 418, 431 Kolmogorov’s theorem, 192 Kronecker product, 30 Kummer’s test, 154

669

INDEX

Lack of fit, 342, 399 Lagrange interpolating polynomial, 411, 416, 423, 518, 521 Lagrange interpolation, 413, 419, 422 accuracy of, 413 Lagrange multipliers, 288, 312, 329, 341, 344, 352, 383 Laguerre polynomials, 451 applications of, 462 Laplace approximation, 531, 546, 548 method of, 531, 534, 546 Law of large numbers, 84 Bernoulli’s, 204 strong, 192 weak, 192 Least-squares estimate, 46, 344 Least-squares polynomial approximation, 453᎐455 Least upper bound, 9, 73 Legendre polynomials, 440, 442 Leibniz’s formula, 127, 443, 449, 452 Level of significance, 15 L’Hospital’s rule, 103, 120, 254 Likelihood equations, 308, 373 function, 308, 373 Limit, 57 left sided, 59 lower, 136 of a multivariable function, 262 one sided, 58 right sided, 59 of a sequence, 133 subsequential, 136 two sided, 58 upper, 136 Limit point, 11, 262 Linearly dependent, 22 Linearly independent, 23 Linear model, 43, 46, 121, 195, 355, 367, 382 balanced, 43 first order, 340 second order, 343 Linear span, 23 Linear transformation, 25 kernel of, 26 one-to-one, 27 Lipschitz condition, 75 Lipschitz continuous function, 75, 409 Logarithmic series distribution, 193 Logistic distribution, 241

Loss function, 86 Lower limit, 136 Lower semicontinuous function, 90 Lower sum, 206, 294 Maclaurin’s integral test, 153 Maclaurin’s series, 111 Mapping, 5 Markov chain Monte Carlo, 549 Markov’s inequality, 19, 185 Matrix, 28 diagonal, 28 full column rank, 34 full rank, 34 full row rank, 34 Hessian, 275, 285, 374 Householder, 38 idempotent, 38 identity, 28 inverse of, 34 Jacobian, 268 moment, 363 orthogonal, 38, 351, 357 partitioned, 30 rank of, 33 singular, 32, 37 skew symmetric, 29 square, 28 symmetric, 29 trace of, 29 transpose of, 29 Maximum absolute, 113, 283, 346 local, 113, 283 relative, 113 Maximum likelihood estimate, 308, 372 Mean, 117, 239 Mean response, 343, 359, 367, 423 maximum of, 343 Mean squared error, 360, 431 integrated, 360, 431 Mean value theorem, 99᎐100, 271 for integrals, 217 Median, 124 Mertens’s theorem, 163 Method of least squares, 121 Minimax estimator, 86 Minimization of prediction variance, 356 Minimum absolute, 113, 283, 336, 346 local, 113, 283 relative, 113 Minimum bias estimation, 398

670 Minimum norm quadratic unbiased estimation, 327, 378 Minkowski’s inequality, 232 Minor, 31 leading principal, 32, 40, 285 principal, 32, 292 Modal approximation, 549 Modulus of continuity, 407 Moment generating function, 186, 250, 506 Moment matrix, 363 Moments, 182, 505 central, 182, 240, 457 design, 362 factorial, 191 first negative, 242 noncentral, 182, 240 region, 361 Monotone function, 76, 101, 210 Monotone sequence, 133 Monte Carlo method, 535, 540 error bound for, 537 variance reduction, 537 Monte Carlo simulation, 83 Multicollinearity, 46, 196, 353, 387 Multimodal function, 336, 338 Multinomial distribution, 325 Multiresponse experiment, 370 function, 370 optimization, 370 Multivariable function, 261 composite, 265, 276 continuous, 264 inverse of, 280 limit of, 262 optima of, 283 partial derivative of, 267 Riemann integral of, 293 Taylor’s theorem for, 277 uniformly continuous, 265 Negative binomial, 182, 195 Neighborhood, 10, 262 deleted, 57, 262 Newton᎐Cotes methods, 523 Newton᎐Raphson method, 331, 373 Noncentral chi-squared, 44 Noncentrality parameter, 44 Nonlinear model, 367 approximate linearization of, 422 design for, 367 Nonlinear parameter, 367 Norm, 179

INDEX

Euclidean, 24, 42, 179, 261 minimum, 380 of a partition, 205 spectral, 179 Normal distribution, 120, 306 bivariate, 311 p-variate, 400 Normal integral, 460, 469 approximation of, 460, 469 Normal random variable, 245 Null space, 26 One-to-one correspondence, 5 One-to-one function, 5 One-way classification model, 387 o notation, 66, 117 O notation, 65, 156 Open set, 10, 262 Optimality of design A, 365 D, 364, 367, 431 E, 365 G, 364 Optimization techniques, 339 Optimum absolute, 113 compromise, 371 conditions, 370 ideal, 371 local, 113, 284 Ordered n-tuple, 4 Ordered pair, 3 Orthogonal complement, 25 Orthogonal design, 358 Orthogonal functions, 437 Orthogonal matrix, 38, 351, 357 parameterization, 49 Orthogonal polynomials, 437, 453 Chebyshev, 444 of the first kind, 444 of the second kind, 445 zeros of, 444 on a finite set, 455 Hermite, 447 applications of, 456 Jacobi, 443 Laguerre, 451 least-squares approximation with, 453 Legendre, 440, 442 sequence of, 437 zeros of, 440 Orthogonal vectors, 24 Orthonormal basis, 24

671

INDEX

Parameterization of orthogonal matrices, 49 Parseval’s theorem, 496 Partial derivative, 267 Partially nonlinear model, 369 Partial sum, 140 Partition, 205, 294 of a set, 5 Periodic extension, 482 function, 473, 489, 504 Periodogram, 501 Piecewise continuous function, 481, 486, 489, 498 Poisson approximation, 194 Poisson distribution, 119, 183, 204 truncated, 373 Poisson process, 122, 131 Poisson random variable, 14, 124, 183 Polynomial approximation, 403 Bernstein, 406, 410 Chebyshev, 444 Hermite, 447, 456, 542 interpolation, 410 Jacobi, 443, 462 Lagrange interpolating, 411, 416, 423 Laguerre, 451, 462 Legendre, 440, 442 piecewise, 418 trigonometric, 495, 500, 504 Power series, 174 distribution, 194 Prediction equation, 348 Prediction variance, 347, 350, 352, 356 minimization of, 356 standardized, 363᎐364 Principal components, 328 Probability of an event, 13 Probability generating function, 190 continuity theorem for, 192 Probability space, 13 Product of matrices, 29 Projection of a vector, 25 Proper subset, 1 Quadratic form, 39 extrema of, 48 nonnegative definite, 40 positive definite, 40 positive semidefinite, 40 Quadrature, 524 Gauss᎐Chebyshev, 526 Gauss᎐Hermite, 528, 542

Gauss᎐Jacobi, 526 Gauss᎐Laguerre, 528 Gauss᎐Legendre, 526 Raabe’s test, 155 Radius of convergence, 174 Random effects, 44 Random variable, 13 absolutely continuous, 116, 239 Bernoulli, 194 binomial, 14, 203 chi-squared, 246 continuous, 14, 82, 116, 239 discrete, 14, 182 function of, 116 normal, 245 Poisson, 14, 124, 183 Random vector, 304 transformations of, 305 Range of a function, 5 Rank of a matrix, 33 Rational number, 8 Ratio test, 148 Rayleigh’s quotient, 41 infimum of, 41 supremum of, 41 Rearrangement of series, 159 Refinement, 206, 214, 294 Region of convergence, 174 Relation, 4 congruence, 5 equivalence, 4 reflexive, 4 symmetric, 4 transitive, 4 Remainder, 110, 257, 279 Response constrained, 352 maximum, 340 minimum, 340 optimum, 350 predicted, 341, 344, 355, 504 primary, 352 surface, 340 Response surface design, 339, 355, 359 Response surface methodology, 339, 355, 367, 504 Response variable, 121 Restricted least squares, 429 Ridge analysis, 343, 349, 352 modified, 350, 354 standard, 354 Ridge estimator, 56, 196

672 Ridge plots, 346 Ridge regression, 195 Riemann integral, 206, 293 double, 295 iterated, 295 n-tuple, 295 Riemann᎐Stieltjes integral, 234 Risk function, 86 Rodrigues formula, 440, 443, 447, 451 Rolle’s theorem, 99, 110 Root test, 149 Rotatability, 347 Saddle point, 114, 284, 344, 349 Saddlepoint approximation, 549 Sample, 14 Sample space, 13 Sample variance, 14 Satterthwaite’s approximation, 324 Scalar, 21 product, 23 Scheffe’s ´ confidence intervals, 382, 384 Schur’s theorem, 42 Sequence, 132, 262 bounded, 132, 262 Cauchy, 138, 222 convergent, 133, 262 divergent, 133 of functions, 165, 227 limit of, 133 limit infimum of, 136, 148, 151 limit supremum of, 136, 148, 151 of matrices, 178 monotone, 133 of orthogonal polynomials, 437 subsequential limit of, 136 uniformly convergent, 166, 169, 228 Series, 140 absolutely convergent, 143, 158, 174 alternating, 158 conditionally convergent, 158 convergent, 141 divergent, 141 Fourier, 471 of functions, 165 geometric, 142 Gram᎐Charlier, 457 harmonic, 145 hypergeometric, 157 Maclaurin’s, 111 of matrices, 178, 180 multiplication of, 162 of positive terms, 144 power, 174

INDEX

rearrangement of, 159 sum of, 140 Taylor’s, 111, 121 trigonometric, 471 uniformly convergent, 166, 169 SetŽs., 1 bounded, 9, 72, 262 closed, 11, 72, 262 compact, 13 complement of, 2 connected, 12 convex, 79 countable, 6 covering of, 12 disconnected, 12 empty, 1 finite, 6 image of, 5 open, 10, 262 partition of, 5 subcovering of, 13 uncountable, 7 universal, 2 Sherman᎐Morrison formula, 54 Sherman᎐Morrison᎐Woodbury formula, 54, 389 Simplex method, 332 Simpson’s method, 521 Simulated annealing, 338 Simultaneous diagonalization, 40 Singularity of a function, 225 Singular-value decomposition, 39, 45 Singular values of a matrix, 46 Size of test, 15 Spectral decomposition theorem, 38, 181, 351 Spherical polar coordinates, 322 Spline functions, 418, 428 approximation, 418, 421 cubic, 419, 428 designs for, 430 error of approximation by, 421 linear, 418 properties of, 418 Squared error loss, 86 Stationary point, 284, 345 Stationary value, 113 Statistic, 14 first-order, 20 Hotelling’s T 2 , 49 nth-order, 20 Steepest ascent method of, 330, 340 path of, 396 Steepest descent

673

INDEX

method of, 329 path of, 343 Step function, 237 Stieltjes moment problem, 185 Stirling’s formula, 532 Stratified sampling, 313 Stratum, 313 Submatrix, 29 leading principal, 29 principal, 29 Subsequence, 135, 262 Subsequential limit, 136 Subset, 1 Supremum, 9, 73 Symmetric difference, 17 Taylor’s formula, 110 Taylor’s series, 111, 121 remainder of, 110, 257, 279 Taylor’s theorem, 108, 114, 277 t distribution, 259 Time series, 500 Topological space, 10 Topology, 10 basis for, 10 Total derivative, 270 Transformation Box and Cox, 120 of continuous random variables, 246 of random vectors, 305 variance stabilizing, 118 Translation invariance, 379 Trapezoidal method, 517 Triangle inequality, 25

Trigonometric polynomialŽs., 495, 500, 504 series, 471 Two-way crossed classification model, 43, 401 Uncountable set, 7 Uniform convergence of sequences, 166, 169, 228 of series, 167, 169 Uniform distribution, 83, 92, 116, 250 Uniformly continuous function, 68, 74, 82, 265, 407 Union, 2 Universal set, 2 Upper limit, 136 Upper semicontinuous function, 90 Upper sum, 206, 294 Variance, 117, 239 Variance components, 44, 378 Variance criterion, 359, 366 Variance stabilizing transformation, 118 Vector, 21 column, 28 row, 28 zero, 21 Vector space, 21 basis for, 23 dimension of, 23 Vector subspace, 22 Voronovsky’s theorem, 410 Weierstrass approximation theorem, 403 Weierstrass’s M-test, 167, 172

WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Peter Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Louise M. Ryan, David W. Scott, Adrian F. M. Smith, Jozef L. Teugels Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall A complete list of the titles in this series appears at the end of this volume.

WILEY SERIES IN PROBABILITY AND STATISTICS ESTABLISHED BY WALTER A. SHEWHART AND SAMUEL S. WILKS Editors: David J. Balding, Peter Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Louise M. Ryan, David W. Scott, Adrian F. M. Smith, Jozef L. Teugels Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall The Wiley Series in Probability and Statistics is well established and authoritative. It covers many topics of current research interest in both pure and applied statistics and probability theory. Written by leading statisticians and institutions, the titles span both state-of-the-art developments in the field and classical methods. Reflecting the wide range of current research in statistics, the series encompasses applied, methodological and theoretical statistics, ranging from applications and new techniques made possible by advances in computerized practice to rigorous treatment of theoretical approaches. This series provides essential and invaluable reading for all statisticians, whether in academia, industry, government, or research.

ABRAHAM and LEDOLTER · Statistical Methods for Forecasting AGRESTI · Analysis of Ordinal Categorical Data AGRESTI · An Introduction to Categorical Data Analysis AGRESTI · Categorical Data Analysis, Second Edition ANDE L · Mathematics of Chance ANDERSON · An Introduction to Multivariate Statistical Analysis, Second Edition *ANDERSON · The Statistical Analysis of Time Series ANDERSON, AUQUIER, HAUCK, OAKES, VANDAELE, and WEISBERG · Statistical Methods for Comparative Studies ANDERSON and LOYNES · The Teaching of Practical Statistics ARMITAGE and DAVID (editors) · Advances in Biometry ARNOLD, BALAKRISHNAN, and NAGARAJA · Records *ARTHANARI and DODGE · Mathematical Programming in Statistics *BAILEY · The Elements of Stochastic Processes with Applications to the Natural Sciences BALAKRISHNAN and KOUTRAS · Runs and Scans with Applications BARNETT · Comparative Statistical Inference, Third Edition BARNETT and LEWIS · Outliers in Statistical Data, Third Edition BARTOSZYNSKI and NIEWIADOMSKA-BUGAJ · Probability and Statistical Inference BASILEVSKY · Statistical Factor Analysis and Related Methods: Theory and Applications BASU and RIGDON · Statistical Methods for the Reliability of Repairable Systems BATES and WATTS · Nonlinear Regression Analysis and Its Applications BECHHOFER, SANTNER, and GOLDSMAN · Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons BELSLEY · Conditioning Diagnostics: Collinearity and Weak Data in Regression BELSLEY, KUH, and WELSCH · Regression Diagnostics: Identifying Influential Data and Sources of Collinearity BENDAT and PIERSOL · Random Data: Analysis and Measurement Procedures, Third Edition

*Now available in a lower priced paperback edition in the Wiley Classics Library.

BERRY, CHALONER, and GEWEKE · Bayesian Analysis in Statistics and Econometrics: Essays in Honor of Arnold Zellner BERNARDO and SMITH · Bayesian Theory BHAT and MILLER · Elements of Applied Stochastic Processes, Third Edition BHATTACHARYA and JOHNSON · Statistical Concepts and Methods BHATTACHARYA and WAYMIRE · Stochastic Processes with Applications BILLINGSLEY · Convergence of Probability Measures, Second Edition BILLINGSLEY · Probability and Measure, Third Edition BIRKES and DODGE · Alternative Methods of Regression BLISCHKE AND MURTHY · Reliability: Modeling, Prediction, and Optimization BLOOMFIELD · Fourier Analysis of Time Series: An Introduction, Second Edition BOLLEN · Structural Equations with Latent Variables BOROVKOV · Ergodicity and Stability of Stochastic Processes BOULEAU · Numerical Methods for Stochastic Processes BOX · Bayesian Inference in Statistical Analysis BOX · R. A. Fisher, the Life of a Scientist BOX and DRAPER · Empirical Model-Building and Response Surfaces *BOX and DRAPER · Evolutionary Operation: A Statistical Method for Process Improvement BOX, HUNTER, and HUNTER · Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building BOX and LUCEÑO · Statistical Control by Monitoring and Feedback Adjustment BRANDIMARTE · Numerical Methods in Finance: A MATLAB-Based Introduction BROWN and HOLLANDER · Statistics: A Biomedical Introduction BRUNNER, DOMHOF, and LANGER · Nonparametric Analysis of Longitudinal Data in Factorial Experiments BUCKLEW · Large Deviation Techniques in Decision, Simulation, and Estimation CAIROLI and DALANG · Sequential Stochastic Optimization CHAN · Time Series: Applications to Finance CHATTERJEE and HADI · Sensitivity Analysis in Linear Regression CHATTERJEE and PRICE · Regression Analysis by Example, Third Edition CHERNICK · Bootstrap Methods: A Practitioner’s Guide CHILÈS and DELFINER · Geostatistics: Modeling Spatial Uncertainty CHOW and LIU · Design and Analysis of Clinical Trials: Concepts and Methodologies CLARKE and DISNEY · Probability and Random Processes: A First Course with Applications, Second Edition *COCHRAN and COX · Experimental Designs, Second Edition CONGDON · Bayesian Statistical Modelling CONOVER · Practical Nonparametric Statistics, Second Edition COOK · Regression Graphics COOK and WEISBERG · Applied Regression Including Computing and Graphics COOK and WEISBERG · An Introduction to Regression Graphics CORNELL · Experiments with Mixtures, Designs, Models, and the Analysis of Mixture Data, Third Edition COVER and THOMAS · Elements of Information Theory COX · A Handbook of Introductory Statistical Methods *COX · Planning of Experiments CRESSIE · Statistics for Spatial Data, Revised Edition ´´ and HORVÁTH · Limit Theorems in Change Point Analysis CSÖRGO DANIEL · Applications of Statistics to Industrial Experimentation DANIEL · Biostatistics: A Foundation for Analysis in the Health Sciences, Sixth Edition *DANIEL · Fitting Equations to Data: Computer Analysis of Multifactor Data, Second Edition *Now available in a lower priced paperback edition in the Wiley Classics Library.

DAVID · Order Statistics, Second Edition *DEGROOT, FIENBERG, and KADANE · Statistics and the Law DEL CASTILLO · Statistical Process Adjustment for Quality Control DETTE and STUDDEN · The Theory of Canonical Moments with Applications in Statistics, Probability, and Analysis DEY and MUKERJEE · Fractional Factorial Plans DILLON and GOLDSTEIN · Multivariate Analysis: Methods and Applications DODGE · Alternative Methods of Regression *DODGE and ROMIG · Sampling Inspection Tables, Second Edition *DOOB · Stochastic Processes DOWDY and WEARDEN · Statistics for Research, Second Edition DRAPER and SMITH · Applied Regression Analysis, Third Edition DRYDEN and MARDIA · Statistical Shape Analysis DUDEWICZ and MISHRA · Modern Mathematical Statistics DUNN and CLARK · Applied Statistics: Analysis of Variance and Regression, Second Edition DUNN and CLARK · Basic Statistics: A Primer for the Biomedical Sciences, Third Edition DUPUIS and ELLIS · A Weak Convergence Approach to the Theory of Large Deviations *ELANDT-JOHNSON and JOHNSON · Survival Models and Data Analysis ETHIER and KURTZ · Markov Processes: Characterization and Convergence EVANS, HASTINGS, and PEACOCK · Statistical Distributions, Third Edition FELLER · An Introduction to Probability Theory and Its Applications, Volume I, Third Edition, Revised; Volume II, Second Edition FISHER and VAN BELLE · Biostatistics: A Methodology for the Health Sciences *FLEISS · The Design and Analysis of Clinical Experiments FLEISS · Statistical Methods for Rates and Proportions, Second Edition FLEMING and HARRINGTON · Counting Processes and Survival Analysis FULLER · Introduction to Statistical Time Series, Second Edition FULLER · Measurement Error Models GALLANT · Nonlinear Statistical Models GHOSH, MUKHOPADHYAY, and SEN · Sequential Estimation GIFI · Nonlinear Multivariate Analysis GLASSERMAN and YAO · Monotone Structure in Discrete-Event Systems GNANADESIKAN · Methods for Statistical Data Analysis of Multivariate Observations, Second Edition GOLDSTEIN and LEWIS · Assessment: Problems, Development, and Statistical Issues GREENWOOD and NIKULIN · A Guide to Chi-Squared Testing GROSS and HARRIS · Fundamentals of Queueing Theory, Third Edition *HAHN · Statistical Models in Engineering HAHN and MEEKER · Statistical Intervals: A Guide for Practitioners HALD · A History of Probability and Statistics and their Applications Before 1750 HALD · A History of Mathematical Statistics from 1750 to 1930 HAMPEL · Robust Statistics: The Approach Based on Influence Functions HANNAN and DEISTLER · The Statistical Theory of Linear Systems HEIBERGER · Computation for the Analysis of Designed Experiments HEDAYAT and SINHA · Design and Inference in Finite Population Sampling HELLER · MACSYMA for Statisticians HINKELMAN and KEMPTHORNE: · Design and Analysis of Experiments, Volume 1: Introduction to Experimental Design HOAGLIN, MOSTELLER, and TUKEY · Exploratory Approach to Analysis of Variance HOAGLIN, MOSTELLER, and TUKEY · Exploring Data Tables, Trends and Shapes *Now available in a lower priced paperback edition in the Wiley Classics Library.

*HOAGLIN, MOSTELLER, and TUKEY · Understanding Robust and Exploratory Data Analysis HOCHBERG and TAMHANE · Multiple Comparison Procedures HOCKING · Methods and Applications of Linear Models: Regression and the Analysis of Variables HOEL · Introduction to Mathematical Statistics, Fifth Edition HOGG and KLUGMAN · Loss Distributions HOLLANDER and WOLFE · Nonparametric Statistical Methods, Second Edition HOSMER and LEMESHOW · Applied Logistic Regression, Second Edition HOSMER and LEMESHOW · Applied Survival Analysis: Regression Modeling of Time to Event Data HØYLAND and RAUSAND · System Reliability Theory: Models and Statistical Methods HUBER · Robust Statistics HUBERTY · Applied Discriminant Analysis HUNT and KENNEDY · Financial Derivatives in Theory and Practice HUSKOVA, BERAN, and DUPAC · Collected Works of Jaroslav Hajek— with Commentary IMAN and CONOVER · A Modern Approach to Statistics JACKSON · A User’s Guide to Principle Components JOHN · Statistical Methods in Engineering and Quality Assurance JOHNSON · Multivariate Statistical Simulation JOHNSON and BALAKRISHNAN · Advances in the Theory and Practice of Statistics: A Volume in Honor of Samuel Kotz JUDGE, GRIFFITHS, HILL, LÜTKEPOHL, and LEE · The Theory and Practice of Econometrics, Second Edition JOHNSON and KOTZ · Distributions in Statistics JOHNSON and KOTZ (editors) · Leading Personalities in Statistical Sciences: From the Seventeenth Century to the Present JOHNSON, KOTZ, and BALAKRISHNAN · Continuous Univariate Distributions, Volume 1, Second Edition JOHNSON, KOTZ, and BALAKRISHNAN · Continuous Univariate Distributions, Volume 2, Second Edition JOHNSON, KOTZ, and BALAKRISHNAN · Discrete Multivariate Distributions JOHNSON, KOTZ, and KEMP · Univariate Discrete Distributions, Second Edition JUREC KOVÁ and SEN · Robust Statistical Procedures: Aymptotics and Interrelations JUREK and MASON · Operator-Limit Distributions in Probability Theory KADANE · Bayesian Methods and Ethics in a Clinical Trial Design KADANE AND SCHUM · A Probabilistic Analysis of the Sacco and Vanzetti Evidence KALBFLEISCH and PRENTICE · The Statistical Analysis of Failure Time Data, Second Edition KASS and VOS · Geometrical Foundations of Asymptotic Inference KAUFMAN and ROUSSEEUW · Finding Groups in Data: An Introduction to Cluster Analysis KEDEM and FOKIANOS · Regression Models for Time Series Analysis KENDALL, BARDEN, CARNE, and LE · Shape and Shape Theory KHURI · Advanced Calculus with Applications in Statistics, Second Edition KHURI, MATHEW, and SINHA · Statistical Tests for Mixed Linear Models KLUGMAN, PANJER, and WILLMOT · Loss Models: From Data to Decisions KLUGMAN, PANJER, and WILLMOT · Solutions Manual to Accompany Loss Models: From Data to Decisions KOTZ, BALAKRISHNAN, and JOHNSON · Continuous Multivariate Distributions, Volume 1, Second Edition KOTZ and JOHNSON (editors) · Encyclopedia of Statistical Sciences: Volumes 1 to 9 with Index *Now available in a lower priced paperback edition in the Wiley Classics Library.

KOTZ and JOHNSON (editors) · Encyclopedia of Statistical Sciences: Supplement Volume KOTZ, READ, and BANKS (editors) · Encyclopedia of Statistical Sciences: Update Volume 1 KOTZ, READ, and BANKS (editors) · Encyclopedia of Statistical Sciences: Update Volume 2 KOVALENKO, KUZNETZOV, and PEGG · Mathematical Theory of Reliability of Time-Dependent Systems with Practical Applications LACHIN · Biostatistical Methods: The Assessment of Relative Risks LAD · Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction LAMPERTI · Probability: A Survey of the Mathematical Theory, Second Edition LANGE, RYAN, BILLARD, BRILLINGER, CONQUEST, and GREENHOUSE · Case Studies in Biometry LARSON · Introduction to Probability Theory and Statistical Inference, Third Edition LAWLESS · Statistical Models and Methods for Lifetime Data LAWSON · Statistical Methods in Spatial Epidemiology LE · Applied Categorical Data Analysis LE · Applied Survival Analysis LEE and WANG · Statistical Methods for Survival Data Analysis, Third Edition LEPAGE and BILLARD · Exploring the Limits of Bootstrap LEYLAND and GOLDSTEIN (editors) · Multilevel Modelling of Health Statistics LIAO · Statistical Group Comparison LINDVALL · Lectures on the Coupling Method LINHART and ZUCCHINI · Model Selection LITTLE and RUBIN · Statistical Analysis with Missing Data, Second Edition LLOYD · The Statistical Analysis of Categorical Data MAGNUS and NEUDECKER · Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition MALLER and ZHOU · Survival Analysis with Long Term Survivors MALLOWS · Design, Data, and Analysis by Some Friends of Cuthbert Daniel MANN, SCHAFER, and SINGPURWALLA · Methods for Statistical Analysis of Reliability and Life Data MANTON, WOODBURY, and TOLLEY · Statistical Applications Using Fuzzy Sets MARDIA and JUPP · Directional Statistics MASON, GUNST, and HESS · Statistical Design and Analysis of Experiments with Applications to Engineering and Science McCULLOCH and SEARLE · Generalized, Linear, and Mixed Models McFADDEN · Management of Data in Clinical Trials McLACHLAN · Discriminant Analysis and Statistical Pattern Recognition McLACHLAN and KRISHNAN · The EM Algorithm and Extensions McLACHLAN and PEEL · Finite Mixture Models McNEIL · Epidemiological Research Methods MEEKER and ESCOBAR · Statistical Methods for Reliability Data MEERSCHAERT and SCHEFFLER · Limit Distributions for Sums of Independent Random Vectors: Heavy Tails in Theory and Practice *MILLER · Survival Analysis, Second Edition MONTGOMERY, PECK, and VINING · Introduction to Linear Regression Analysis, Third Edition MORGENTHALER and TUKEY · Configural Polysampling: A Route to Practical Robustness MUIRHEAD · Aspects of Multivariate Statistical Theory MURRAY · X-STAT 2.0 Statistical Experimentation, Design Data Analysis, and Nonlinear Optimization *Now available in a lower priced paperback edition in the Wiley Classics Library.

MYERS and MONTGOMERY · Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Second Edition MYERS, MONTGOMERY, and VINING · Generalized Linear Models. With Applications in Engineering and the Sciences NELSON · Accelerated Testing, Statistical Models, Test Plans, and Data Analyses NELSON · Applied Life Data Analysis NEWMAN · Biostatistical Methods in Epidemiology OCHI · Applied Probability and Stochastic Processes in Engineering and Physical Sciences OKABE, BOOTS, SUGIHARA, and CHIU · Spatial Tesselations: Concepts and Applications of Voronoi Diagrams, Second Edition OLIVER and SMITH · Influence Diagrams, Belief Nets and Decision Analysis PANKRATZ · Forecasting with Dynamic Regression Models PANKRATZ · Forecasting with Univariate Box-Jenkins Models: Concepts and Cases *PARZEN · Modern Probability Theory and Its Applications PEÑA, TIAO, and TSAY · A Course in Time Series Analysis PIANTADOSI · Clinical Trials: A Methodologic Perspective PORT · Theoretical Probability for Applications POURAHMADI · Foundations of Time Series Analysis and Prediction Theory PRESS · Bayesian Statistics: Principles, Models, and Applications PRESS and TANUR · The Subjectivity of Scientists and the Bayesian Approach PUKELSHEIM · Optimal Experimental Design PURI, VILAPLANA, and WERTZ · New Perspectives in Theoretical and Applied Statistics PUTERMAN · Markov Decision Processes: Discrete Stochastic Dynamic Programming *RAO · Linear Statistical Inference and Its Applications, Second Edition RENCHER · Linear Models in Statistics RENCHER · Methods of Multivariate Analysis, Second Edition RENCHER · Multivariate Statistical Inference with Applications RIPLEY · Spatial Statistics RIPLEY · Stochastic Simulation ROBINSON · Practical Strategies for Experimenting ROHATGI and SALEH · An Introduction to Probability and Statistics, Second Edition ROLSKI, SCHMIDLI, SCHMIDT, and TEUGELS · Stochastic Processes for Insurance and Finance ROSENBERGER and LACHIN · Randomization in Clinical Trials: Theory and Practice ROSS · Introduction to Probability and Statistics for Engineers and Scientists ROUSSEEUW and LEROY · Robust Regression and Outlier Detection RUBIN · Multiple Imputation for Nonresponse in Surveys RUBINSTEIN · Simulation and the Monte Carlo Method RUBINSTEIN and MELAMED · Modern Simulation and Modeling RYAN · Modern Regression Methods RYAN · Statistical Methods for Quality Improvement, Second Edition SALTELLI, CHAN, and SCOTT (editors) · Sensitivity Analysis *SCHEFFE · The Analysis of Variance SCHIMEK · Smoothing and Regression: Approaches, Computation, and Application SCHOTT · Matrix Analysis for Statistics SCHUSS · Theory and Applications of Stochastic Differential Equations SCOTT · Multivariate Density Estimation: Theory, Practice, and Visualization *SEARLE · Linear Models SEARLE · Linear Models for Unbalanced Data SEARLE · Matrix Algebra Useful for Statistics SEARLE, CASELLA, and McCULLOCH · Variance Components SEARLE and WILLETT · Matrix Algebra for Applied Economics SEBER · Linear Regression Analysis *Now available in a lower priced paperback edition in the Wiley Classics Library.

SEBER · Multivariate Observations SEBER and WILD · Nonlinear Regression SENNOTT · Stochastic Dynamic Programming and the Control of Queueing Systems *SERFLING · Approximation Theorems of Mathematical Statistics SHAFER and VOVK · Probability and Finance: It’s Only a Game! SMALL and MCLEISH · Hilbert Space Methods in Probability and Statistical Inference SRIVASTAVA · Methods of Multivariate Statistics STAPLETON · Linear Statistical Models STAUDTE and SHEATHER · Robust Estimation and Testing STOYAN, KENDALL, and MECKE · Stochastic Geometry and Its Applications, Second Edition STOYAN and STOYAN · Fractals, Random Shapes and Point Fields: Methods of Geometrical Statistics STYAN · The Collected Papers of T. W. Anderson: 1943–1985 SUTTON, ABRAMS, JONES, SHELDON, and SONG · Methods for Meta-Analysis in Medical Research TANAKA · Time Series Analysis: Nonstationary and Noninvertible Distribution Theory THOMPSON · Empirical Model Building THOMPSON · Sampling, Second Edition THOMPSON · Simulation: A Modeler’s Approach THOMPSON and SEBER · Adaptive Sampling THOMPSON, WILLIAMS, and FINDLAY · Models for Investors in Real World Markets TIAO, BISGAARD, HILL, PEÑA, and STIGLER (editors) · Box on Quality and Discovery: with Design, Control, and Robustness TIERNEY · LISP-STAT: An Object-Oriented Environment for Statistical Computing and Dynamic Graphics TSAY · Analysis of Financial Time Series UPTON and FINGLETON · Spatial Data Analysis by Example, Volume II: Categorical and Directional Data VAN BELLE · Statistical Rules of Thumb VIDAKOVIC · Statistical Modeling by Wavelets WEISBERG · Applied Linear Regression, Second Edition WELSH · Aspects of Statistical Inference WESTFALL and YOUNG · Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment WHITTAKER · Graphical Models in Applied Multivariate Statistics WINKER · Optimization Heuristics in Economics: Applications of Threshold Accepting WONNACOTT and WONNACOTT · Econometrics, Second Edition WOODING · Planning Pharmaceutical Clinical Trials: Basic Statistical Principles WOOLSON and CLARKE · Statistical Methods for the Analysis of Biomedical Data, Second Edition WU and HAMADA · Experiments: Planning, Analysis, and Parameter Design Optimization YANG · The Construction Theory of Denumerable Markov Processes *ZELLNER · An Introduction to Bayesian Inference in Econometrics ZHOU, OBUCHOWSKI, and MCCLISH · Statistical Methods in Diagnostic Medicine

*Now available in a lower priced paperback edition in the Wiley Classics Library.