TABLE OF CONTENTS
✦ ✦ ✦ ✦
Table of Contents Preface Chapter 1.1. 1.2. 1.3. 1.4. 1.5. 1.6. 1.7. 1.8.
ix 1. Computer S...
1274 downloads
6345 Views
6MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
TABLE OF CONTENTS
✦ ✦ ✦ ✦
Table of Contents Preface Chapter 1.1. 1.2. 1.3. 1.4. 1.5. 1.6. 1.7. 1.8.
ix 1. Computer Science: The Mechanization of Abstraction What This Book Is About 3 What This Chapter Is About 6 Data Models 6 The C Data Model 13 Algorithms and the Design of Programs 20 Some C Conventions Used Throughout the Book 22 Summary of Chapter 1 23 Bibliographic Notes for Chapter 1 24
Chapter 2. Iteration, Induction, and Recursion 25 2.1. What This Chapter Is About 27 2.2. Iteration 27 2.3. Inductive Proofs 34 2.4. Complete Induction 44 2.5. Proving Properties of Programs 52 2.6. Recursive Definitions 59 2.7. Recursive Functions 69 2.8. Merge Sort: A Recursive Sorting Algorithm 75 2.9. Proving Properties of Recursive Programs 84 2.10. Summary of Chapter 2 87 2.11. Bibliographic Notes for Chapter 2 88 Chapter 3. The Running Time of Programs 89 3.1. What This Chapter Is About 89 3.2. Choosing an Algorithm 90 3.3. Measuring Running Time 91 3.4. Big-Oh and Approximate Running Time 96 3.5. Simplifying Big-Oh Expressions 101 3.6. Analyzing the Running Time of a Program 109 3.7. A Recursive Rule for Bounding Running Time 116 3.8. Analyzing Programs with Function Calls 127 3.9. Analyzing Recursive Functions 132 3.10. Analysis of Merge Sort 136 3.11. Solving Recurrence Relations 144 3.12. Summary of Chapter 3 154 3.13. Bibliographic Notes for Chapter 3 155 Chapter 4.1. 4.2. 4.3. 4.4.
4. Combinatorics and Probability 156 What This Chapter Is About 156 Counting Assignments 157 Counting Permutations 160 Ordered Selections 167
1
v
vi
TABLE OF CONTENTS
4.5. Unordered Selections 170 4.6. Orderings With Identical Items 178 4.7. Distribution of Objects to Bins 181 4.8. Combining Counting Rules 184 4.9. Introduction to Probability Theory 187 4.10. Conditional Probability 193 4.11. Probabilistic Reasoning 203 4.12. Expected Value Calculations 212 4.13. Some Programming Applications of Probability 4.14. Summary of Chapter 4 220 4.15. Bibliographic Notes for Chapter 4 221 Chapter 5. The Tree Data Model 223 5.1. What This Chapter Is About 223 5.2. Basic Terminology 224 5.3. Data Structures for Trees 231 5.4. Recursions on Trees 239 5.5. Structural Induction 248 5.6. Binary Trees 253 5.7. Binary Search Trees 258 5.8. Efficiency of Binary Search Tree Operations 268 5.9. Priority Queues and Partially Ordered Trees 271 5.10. Heapsort: Sorting with Balanced POTs 280 5.11. Summary of Chapter 5 284 5.12. Bibliographic Notes for Chapter 5 285 Chapter 6. The List Data Model 286 6.1. What This Chapter Is About 286 6.2. Basic Terminology 287 6.3. Operations on Lists 291 6.4. The Linked-List Data Structure 293 6.5. Array-Based Implementation of Lists 301 6.6. Stacks 306 6.7. Implementing Function Calls Using a Stack 6.8. Queues 318 6.9. Longest Common Subsequences 321 6.10. Representing Character Strings 327 6.11. Summary of Chapter 6 334 6.12. Bibliographic Notes for Chapter 6 335 Chapter 7.1. 7.2. 7.3. 7.4. 7.5. 7.6. 7.7. 7.8. 7.9.
7. The Set Data Model 337 What This Chapter Is About 337 Basic Definitions 338 Operations on Sets 342 List Implementation of Sets 351 Characteristic-Vector Implementation of Sets Hashing 360 Relations and Functions 366 Implementing Functions as Data 373 Implementing Binary Relations 380
312
357
215
TABLE OF CONTENTS
7.10. 7.11. 7.12. 7.13.
Some Special Properties of Binary Relations Infinite Sets 396 Summary of Chapter 7 401 Bibliographic Notes for Chapter 7 402
386
Chapter 8. The Relational Data Model 403 8.1. What This Chapter Is About 403 8.2. Relations 404 8.3. Keys 411 8.4. Primary Storage Structures for Relations 414 8.5. Secondary Index Structures 419 8.6. Navigation among Relations 423 8.7. An Algebra of Relations 428 8.8. Implementing Relational Algebra Operations 436 8.9. Algebraic Laws for Relations 440 8.10. Summary of Chapter 8 449 8.11. Bibliographic Notes for Chapter 8 450 Chapter 9. The Graph Data Model 451 9.1. What This Chapter Is About 451 9.2. Basic Concepts 452 9.3. Implementation of Graphs 459 9.4. Connected Components of an Undirected Graph 9.5. Minimal Spanning Trees 478 9.6. Depth-First Search 484 9.7. Some Uses of Depth-First Search 495 9.8. Dijkstra’s Algorithm for Finding Shortest Paths 9.9. Floyd’s Algorithm for Shortest Paths 513 9.10. An Introduction to Graph Theory 521 9.11. Summary of Chapter 9 526 9.12. Bibliographic Notes for Chapter 9 527
466
502
Chapter 10. Patterns, Automata, and Regular Expressions 529 10.1. What This Chapter Is About 530 10.2. State Machines and Automata 530 10.3. Deterministic and Nondeterministic Automata 536 10.4. From Nondeterminism to Determinism 547 10.5. Regular Expressions 556 10.6. The UNIX Extensions to Regular Expressions 564 10.7. Algebraic Laws for Regular Expressions 568 10.8. From Regular Expressions to Automata 571 10.9. From Automata to Regular Expressions 582 10.10. Summary of Chapter 10 588 10.11. Bibliographic Notes for Chapter 10 589 Chapter 11. Recursive Description of Patterns 11.1. What This Chapter Is About 591 11.2. Context-Free Grammars 592 11.3. Languages from Grammars 599 11.4. Parse Trees 602
591
vii
viii
TABLE OF CONTENTS
11.5. Ambiguity and the Design of Grammars 610 11.6. Constructing Parse Trees 617 11.7. A Table-Driven Parsing Algorithm 625 11.8. Grammars Versus Regular Expressions 634 11.9. Summary of Chapter 11 640 11.10. Bibliographic Notes for Chapter 11 641 Chapter 12. Propositional Logic 642 12.1. What This Chapter Is About 642 12.2. What Is Propositional Logic? 643 12.3. Logical Expressions 645 12.4. Truth Tables 649 12.5. From Boolean Functions to Logical Expressions 655 12.6. Designing Logical Expressions by Karnaugh Maps 660 12.7. Tautologies 669 12.8. Some Algebraic Laws for Logical Expressions 674 12.9. Tautologies and Methods of Proof 682 12.10. Deduction 686 12.11. Proofs by Resolution 692 12.12. Summary of Chapter 12 697 12.13. Bibliographic Notes for Chapter 12 698 Chapter 13. Using Logic to Design Computer Components 13.1. What This Chapter is About 699 13.2. Gates 700 13.3. Circuits 701 13.4. Logical Expressions and Circuits 705 13.5. Some Physical Constraints on Circuits 711 13.6. A Divide-and-Conquer Addition Circuit 716 13.7. Design of a Multiplexer 723 13.8. Memory Elements 730 13.9. Summary of Chapter 13 731 13.10. Bibliographic Notes for Chapter 13 732 Chapter 14. Predicate Logic 733 14.1. What This Chapter Is About 733 14.2. Predicates 734 14.3. Logical Expressions 736 14.4. Quantifiers 739 14.5. Interpretations 745 14.6. Tautologies 751 14.7. Tautologies Involving Quantifiers 753 14.8. Proofs in Predicate Logic 759 14.9. Proofs from Rules and Facts 762 14.10. Truth and Provability 768 14.11. Summary of Chapter 14 774 14.12. Bibliographic Notes for Chapter 14 775 Index
776
699
Preface This book was motivated by the desire we and others have had to further the evolution of the core course in computer science. Many departments across the country have revised their curriculum in response to the introductory course in the science of computing discussed in the “Denning Report,” (Denning, P. J., D. E. Comer, D. Gries, M. C. Mulder, A. Tucker, J. Turner, and P. R. Young, “Computing as a Discipline,” Comm. ACM 32:1, pp. 9–23, January 1989.). That report draws attention to three working methodologies or processes — theory, abstraction, and design — as fundamental to all undergraduate programs in the discipline. More recently, the Computing Curricula 1991 report of the joint ACM/IEEE-CS Curriculum Task Force echoes the Denning Report in identifying key recurring concepts which are fundamental to computing, especially: conceptual and formal models, efficiency, and levels of abstraction. The themes of these two reports summarize what we have tried to offer the student in this book. This book developed from notes for a two-quarter course at Stanford — called CS109: Introduction to Computer Science — that serves a number of goals. The first goal is to give beginning computer science majors a solid foundation for further study. However, computing is becoming increasingly important in a much wider range of scientific and engineering disciplines. Therefore, a second goal is to give those students who will not take advanced courses in computer science the conceptual tools that the field provides. Finally, a more pervasive goal is to expose all students not only to programming concepts but also to the intellectually rich foundations of the field. Our first version of this book was based on programming in Pascal and appeared in 1992. Our choice of Pascal as the language for example programs was motivated by that language’s use in the Computer Science Advanced Placement Test as well as in a plurality of college introductions to programming. We were pleased to see that since 1992 there has been a significant trend toward C as the introductory programming language, and we accordingly developed a new version of the book using C for programming examples. Our emphasis on abstraction and encapsulation should provide a good foundation for subsequent courses covering object-oriented technology using C++. At the same time, we decided to make two significant improvements in the content of the book. First, although it is useful to have a grounding in machine architecture to motivate how running time is measured, we found that almost all curricula separate architecture into a separate course, so the chapter on that subject was not useful. Second, many introductory courses in the theory of computing emphasize combinatorics and probability, so we decided to increase the coverage and cluster the material into a chapter of its own. Foundations of Computer Science covers subjects that are often found split between a discrete mathematics course and a sophomore-level sequence in computer science in data structures. It has been our intention to select the mathematical foundations with an eye toward what the computer user really needs, rather than what a mathematician might choose. We have tried to integrate effectively the mathematical foundations with the computing. We thus hope to provide a better feel for the soul of computer science than might be found in a programming course, ix
x
PREFACE
a discrete mathematics course, or a course in a computer science subspecialty. We believe that, as time goes on, all scientists and engineers will take a foundational course similar to the one offered at Stanford upon which this book is based. Such a course in computer science should become as standard as similar courses in calculus and physics.
Prerequisites Students taking courses based on this book have ranged from first-year undergraduates to graduate students. We assume only that students have had a solid course in programming. They should be familiar with the programming language ANSI C to use this edition. In particular, we expect students to be comfortable with C constructs such as recursive functions, structures, pointers, and operators involving pointers and structures such as dot, ->, and &.
Suggested Outlines for Foundational Courses in CS In terms of a traditional computer science curriculum, the book combines a first course in data structures — that is, a “CS2” course — with a course in discrete mathematics. We believe that the integration of these subjects is extremely desirable for two reasons: 1.
It helps motivate the mathematics by relating it more closely to the computing.
2.
Computing and mathematics can be mutually reinforcing. Some examples are the way recursive programming and mathematical induction are related in Chapter 2 and the way the free/bound variable distinction for logic is related to the scope of variables in programming languages in Chapter 14. Suggestions for instructive programming assignments are presented throughout the book.
There are a number of ways in which this book can be used.
A Two-Quarter or Two-Semester Course The CS109A-B sequence at Stanford is typical of what might be done in two quarters, although these courses are rather intensive, being 4-unit, 10-week courses each. These two courses cover the entire book, the first seven chapters in CS109A and Chapters 8 through 14 in CS109B.
A One-Semester “CS2” Type Course It is possible to use the book for a one-semester course covering a set of topics similar to what would appear in a “CS2” course. Naturally, there is too much material in the book to cover in one semester, and so we recommend the following: 1.
Recursive algorithms and programs in Sections 2.7 and 2.8.
2.
Big-oh analysis and running time of programs: all of Chapter 3 except for Section 3.11 on solving recurrence relations.
3.
Trees in Sections 5.2 through 5.10.
PREFACE
xi
4.
Lists: all of Chapter 6. Some may wish to cover lists before trees, which is a more traditional treatment. We regard trees as the more fundamental notion, but there is little harm in switching the order. The only significant dependency is that Chapter 6 talks about the “dictionary” abstract data type (set with operations insert, delete, and lookup), which is introduced in Section 5.7 as a concept in connection with binary search trees.
5.
Sets and relations. Data structures for sets and relations are emphasized in Sections 7.2 through 7.9 and 8.2 through 8.6.
6.
Graph algorithms are covered in Sections 9.2 through 9.9.
A One-Semester Discrete Mathematics Course For a one-semester course emphasizing mathematical foundations, the instructor could choose to cover: 1. 2.
Mathematical induction and recursive programs in Chapter 2. Big-oh analysis, running time, and recurrence relations in Sections 3.4 through 3.11. 3. Combinatorics in Sections 4.2 through 4.8. 4. Discrete probability in Sections 4.9 through 4.13. 5. Mathematical aspects of trees in Sections 5.2 through 5.6. 6. Mathematical aspects of sets in Sections 7.2, 7.3, 7.7, 7.10, and 7.11. 7. The algebra of relations in Sections 8.2, 8.7, and 8.9. 8. Graph algorithms and graph theory in Chapter 9. 9. Automata and regular expressions in Chapter 10. 10. Context-free grammars in Sections 11.2 through 11.4. 11. Propositional and predicate logic in Chapters 12 and 14, respectively.
Features of This Book To help the student assimilate the material, we have added the following study aids: 1.
Each chapter has an outline section at the beginning and a summary section at the end highlighting the main points.
2.
Marginal notes mark important concepts and definitions. However, items mentioned in section or subsection headlines are not repeated in the margin.
3.
“Sidebars” are separated from the text by double lines. These short notes serve several purposes: ✦
Some are elaborations on the text or make some fine points about program or algorithm design.
✦
Others are for summary or emphasis of points made in the text nearby. These include outlines of important kinds of proofs, such as the various forms of proof by induction.
✦
A few are used to give examples of fallacious arguments, and we hope that the separation from the text in this way will eliminate possible misconstruction of the point.
xii
PREFACE
✦
A few give very brief introductions to major topics like undecidability or the history of computers to which we wish we could devote a full section.
4.
Most of the sections end with exercises. There are more than 1000 exercises or parts spread among the sections. Of these roughly 30% are marked with a single star, which indicates that they require more thought than the unstarred exercises. Approximately another 10% of the exercises are doubly starred, and these are the most challenging.
5.
Chapters end with bibliographic notes. We have not attempted to be exhaustive, but offer suggestions for more advanced texts on the subject of the chapter and mention the relevant papers with the most historical significance.
About the Cover It is a tradition for computer science texts to have a cover with a cartoon or drawing symbolizing the content of the book. Here, we have drawn on the myth of the world as the back of a turtle, but our world is populated with representatives of some of the other, more advanced texts in computer science that this book is intended to support. They are: The teddy bear: R. Sethi, Programming Languages: Concepts and Constructs, Addison-Wesley, Reading, Mass., 1989. The baseball player: J. D. Ullman, Principles of Database and Knowledge-Base Systems, Computer Science Press, New York, 1988. The column: J. L. Hennessy and D. A. Patterson, Computer Architecture: a Quantitative Approach, Morgan-Kaufmann, San Mateo, Calif., 1990. The dragon: A. V. Aho, R. Sethi, and J. D. Ullman, Compiler Design: Principles, Techniques, and Tools, Addison-Wesley, Reading, Mass., 1986. The triceratops: J. L. Peterson and A. Silberschatz, Operating Systems Concepts, second edition, Addison-Wesley, Reading, Mass., 1985.
Acknowledgments We are deeply indebted to a number of colleagues and students who have read this material and given us many valuable suggestions for improving the presentation. We owe a special debt of gratitude to Brian Kernighan, Don Knuth, Apostolos Lerios, and Bob Martin who read the original Pascal manuscript in detail and gave us many perceptive comments. We have received, and gratefully acknowledge, reports of course testing of the notes for the Pascal edition of this book by Michael Anderson, Margaret Johnson, Udi Manber, Joseph Naor, Prabhakar Ragde, Rocky Ross, and Shuky Sagiv. There are a number of other people who found errors in earlier editions, both the original notes and the various printings of the Pascal edition. In this regard, we would like to thank: Susan Aho, Michael Anderson, Aaron Edsinger, Lonnie Eldridge, Todd Feldman, Steve Friedland, Christopher Fuselier, Mike Genstil, Paul Grubb III, Barry Hayes, John Hwang, Hakan Jakobsson, Arthur Keller, Dean Kelley, James Kuffner Jr., Steve Lindell, Richard Long, Mark MacDonald, Simone Martini, Hugh McGuire, Alan Morgan, Monnia Oropeza, Rodrigo Philander, Andrew Quan, Stuart Reges, John Stone, Keith Swanson, Steve Swenson, Sanjai Tiwari, Eric Traut, and Lynzi Ziegenhagen.
PREFACE
xiii
We acknowledge helpful advice from Geoff Clem, Jon Kettenring, and Brian Kernighan during the preparation of the C edition of Foundations of Computer Science. Peter Ullman produced a number of the figures used in this book. We are grateful to Dan Clayton, Anthony Dayao, Mat Howard, and Ron Underwood for help with TEX fonts, and to Hester Glynn and Anne Smith for help with the manuscript preparation.
On-Line Access to Code, Errata, and Notes You can obtain copies of the major programs in this book by anonymous ftp to host ftp-cs.stanford.edu. Login with user name anonymous and give your name and host as a password. You may then execute cd fcsc where you will find programs from this book. We also plan to keep in this directory information about errata and what course notes we can provide. A. V. A. Chatham, NJ J. D. U. Stanford, CA July, 1994
CHAPTER
✦ ✦ ✦ ✦
Abstraction
Exam scheduling
1 Computer Science: The Mechanization of Abstraction Though it is a new field, computer science already touches virtually every aspect of human endeavor. Its impact on society is seen in the proliferation of computers, information systems, text editors, spreadsheets, and all of the wonderful application programs that have been developed to make computers easier to use and people more productive. An important part of the field deals with how to make programming easier and software more reliable. But fundamentally, computer science is a science of abstraction — creating the right model for thinking about a problem and devising the appropriate mechanizable techniques to solve it. Every other science deals with the universe as it is. The physicist’s job, for example, is to understand how the world works, not to invent a world in which physical laws would be simpler or more pleasant to follow. Computer scientists, on the other hand, must create abstractions of real-world problems that can be understood by computer users and, at the same time, that can be represented and manipulated inside a computer. Sometimes the process of abstraction is simple. For example, we can model the behavior of the electronic circuits used to build computers quite well by an abstraction called “propositional logic.” The modeling of circuits by logical expressions is not exact; it simplifies, or abstracts away, many details — such as the time it takes for electrons to flow through circuits and gates. Nevertheless, the propositional logic model is good enough to help us design computer circuits well. We shall have much more to say about propositional logic in Chapter 12. As another example, suppose we are faced with the problem of scheduling final examinations for courses. That is, we must assign course exams to time slots so that two courses may have their exams scheduled in the same time slot only if there is no student taking both. At first, it may not be apparent how we should model this problem. One approach is to draw a circle called a node for each course and draw a line called an edge connecting two nodes if the corresponding courses have a student in common. Figure 1.1 suggests a possible picture for five courses; the picture is called a course-conflict graph. Given the course-conflict graph, we can solve the exam-scheduling problem by repeatedly finding and removing “maximal independent sets” from the graph. An 1
2
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
Eng
Math
CS
Econ
Phy
Fig. 1.1. Course-conflict graph for five courses. An edge between two courses indicates that at least one student is taking both courses.
Maximal independent set
independent set is a collection of nodes that have no connecting edges within the collection. An independent set is maximal if no other node from the graph can be added without including an edge between two nodes of the set. In terms of courses, a maximal independent set is any maximal set of courses with no common students. In Fig. 1.1, {Econ, Eng, P hy} is one maximal independent set. The set of courses corresponding to the selected maximal independent set is assigned to the first time slot. We remove from the graph the nodes in the first maximal independent set, along with all incident edges, and then find a maximal independent set among the remaining courses. One choice for the next maximal independent set is the singleton set {CS}. The course in this maximal independent set is assigned to the second time slot. We repeat this process of finding and deleting maximal independent sets until no more nodes remain in the course-conflict graph. At this point, all courses will have been assigned to time slots. In our example, after two iterations, the only remaining node in the course-conflict graph is Math, and this forms the final maximal independent set, which is assigned to the third time slot. The resulting exam schedule is thus TIME SLOT 1 2 3
COURSE EXAMS Econ, Eng, Phy CS Math
This algorithm does not necessarily partition the courses among the smallest possible number of time slots, but it is simple and does tend to produce a schedule with close to the smallest number of time slots. It is also one that can be readily programmed using the techniques presented in Chapter 9. Notice that this approach abstracts away some details of the problem that may be important. For example, it could cause one student to have five exams in five consecutive time slots. We could create a model that included limits on how many exams in a row one student could take, but then both the model and the solution
SEC. 1.1
WHAT THIS BOOK IS ABOUT
3
Abstraction: Not to Be Feared The reader may cringe at the word “abstraction,” because we all have the intuition that abstract things are hard to understand; for example, abstract algebra (the study of groups, rings, and the like) is generally considered harder than the algebra we learned in high school. However, abstraction in the sense we use it implies simplification, the replacement of a complex and detailed real-world situation by an understandable model within which we can solve a problem. That is, we “abstract away” the details whose effect on the solution to a problem is minimal or nonexistent, thereby creating a model that lets us deal with the essence of the problem.
Knowledge representation
✦ ✦ ✦ ✦
1.1
to the exam-scheduling problem would be more complicated. Often, finding a good abstraction can be quite difficult because we are forced to confront the fundamental limitations on the tasks computers can perform and the speed with which computers can perform those tasks. In the early days of computer science, some optimists believed that robots would soon have the prodigious capability and versatility of the Star Wars robot C3PO. Since then we have learned that in order to have “intelligent” behavior on the part of a computer (or robot), we need to provide that computer with a model of the world that is essentially as detailed as that possessed by humans, including not only facts (“Sally’s phone number is 555-1234”), but principles and relationships (“If you drop something, it usually falls downward”). We have made much progress on this problem of “knowledge representation.” We have devised abstractions that can be used to help build programs that do certain kinds of reasoning. One example of such an abstraction is the directed graph, in which nodes represent entities (“the species cat” or “Fluffy”) and arrows (called arcs) from one node to another represent relationships (“Fluffy is a cat,” “cats are animals,” “Fluffy owns Fluffy’s milk saucer”); Figure 1.2 suggests such a graph. Another useful abstraction is formal logic, which allows us to manipulate facts by applying rules of inference, such as “If X is a cat and Y is the mother of X, then Y is a cat.” Nevertheless, progress on modeling, or abstracting, the real world or significant pieces thereof remains a fundamental challenge of computer science, one that is not likely to be solved completely in the near future.
What This Book Is About This book will introduce the reader, who is assumed to have a working knowledge of the programming language ANSI C, to the principal ideas and concerns of computer science. The book emphasizes three important problem-solving tools: 1.
Data models, the abstractions used to describe problems. We have already mentioned two models: logic and graphs. We shall meet many others throughout this book.
4
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
Animal
is
Cat
is
Fluffy
owns
Fluffy’s milk saucer
Fig. 1.2. A graph representing knowledge about Fluffy.
2.
Data structures, the programming-language constructs used to represent data models. For example, C provides built-in abstractions, such as structures and pointers, that allow us to construct data structures to represent complex abstractions such as graphs.
3.
Algorithms, the techniques used to obtain solutions by manipulating data as represented by the abstractions of a data model, by data structures, or by other means.
Data Models We meet data models in two contexts. Data models such as the graphs discussed in the introduction to this chapter are abstractions frequently used to help formulate solutions to problems. We shall learn about several such data models in this book: trees in Chapter 5, lists in Chapter 6, sets in Chapter 7, relations in Chapter 8, graphs in Chapter 9, finite automata in Chapter 10, grammars in Chapter 11, and logic in Chapters 12 and 14. Data models are also associated with programming languages and computers. For example, C has a data model that includes abstractions such as characters, integers of several sizes, and floating-point numbers. Integers and floating-point numbers in C are only approximations of integers and reals in mathematics because of the limited precision of arithmetic available in computers. The C data model also includes types such as structures, pointers, and functions, which we shall discuss in more detail in Section 1.4.
Data Structures When the data model of the language in which we are writing a program lacks a built-in representation for the data model of the problem at hand, we must represent the needed data model using the abstractions supported by the language. For this purpose, we study data structures, which are methods for representing in the data model of a programming language abstractions that are not an explicit part of
SEC. 1.1
WHAT THIS BOOK IS ABOUT
5
that language. Different programming languages may have strikingly different data models. For example, unlike C, the language Lisp supports trees directly, and the language Prolog has logic built into its data model.
Algorithms An algorithm is a precise and unambiguous specification of a sequence of steps that can be carried out mechanically. The notation in which an algorithm is expressed can be any commonly understood language, but in computer science algorithms are most often expressed formally as programs in a programming language, or in an informal style as a sequence of programming language constructs intermingled with English language statements. Most likely, you have already encountered several important algorithms while studying programming. For example, there are a number of algorithms for sorting the elements of an array, that is, putting the elements in smallest-first order. There are clever searching algorithms such as binary search, which quickly finds a given element in a sorted array by repeatedly dividing in half the portion of the array in which the element could appear. These, and many other “tricks” for solving common problems, are among the tools the computer scientist uses when designing programs. We shall study many such techniques in this book, including the important methods for sorting and searching. In addition, we shall learn what makes one algorithm better than another. Frequently, the running time, or time taken by an algorithm measured as a function of the size of its input, is one important aspect of the “quality” of the algorithm; we discuss running time in Chapter 3. Other aspects of algorithms are also important, particularly their simplicity. Ideally, an algorithm should be easy to understand and easy to turn into a working program. Also, the resulting program should be understandable by a person reading the code that implements the algorithm. Unfortunately, our desires for a fast algorithm and a simple algorithm are often in conflict, and we must choose our algorithm wisely.
Underlying Threads As we progress through this book, we shall encounter a number of important unifying principles. We alert the reader to two of these here: 1.
Design algebras. In certain fields in which the underlying models have become well understood, we can develop notations in which design trade-offs can be expressed and evaluated. Through this understanding, we can develop a theory of design with which well-engineered systems can be constructed. Propositional logic, with the associated notation called Boolean algebra that we encounter in Chapter 12, is a good example of this kind of design algebra. With it, we can design efficient circuits for subsystems of the kind found in digital computers. Other examples of algebras found in this book are the algebra of sets in Chapter 7, the algebra of relations in Chapter 8, and the algebra of regular expressions in Chapter 10.
6
2.
✦ ✦ ✦ ✦
1.2
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
Recursion is such a useful technique for defining concepts and solving problems that it deserves special mention. We discuss recursion in detail in Chapter 2 and use it throughout the rest of the book. Whenever we need to define an object precisely or whenever we need to solve a problem, we should always ask, “What does the recursive solution look like?” Frequently that solution has a simplicity and efficiency that makes it the method of choice.
What This Chapter Is About The remainder of this chapter sets the stage for the study of computer science. The primary concepts that will be covered are ✦
Data models (Section 1.3)
✦
The data model of the programming language C (Section 1.4)
✦
The principal steps in the software-creation process (Section 1.5)
We shall give examples of several different ways in which abstractions and models appear in computer systems. In particular, we mention the models found in programming languages, in certain kinds of systems programs, such as operating systems, and in the circuits from which computers are built. Since software is a vital component of today’s computer systems, we need to understand the softwarecreation process, the role played by models and algorithms, and the aspects of software creation that computer science can address only in limited ways. In Section 1.6 there are some conventional definitions that are used in C programs throughout this book. ✦ ✦ ✦ ✦
1.3
Data Models Any mathematical concept can be termed a data model. In computer science, a data model normally has two aspects: 1.
The values that objects can assume. For example, many data models contain objects that have integer values. This aspect of the data model is static; it tells us what values objects may take. The static part of a programming language’s data model is often called the type system.
2.
The operations on the data. For example, we normally apply operations such as addition to integers. This aspect of the model is dynamic; it tells us the ways in which we can change values and create new values.
Type system
Programming Language Data Models
Data object
Each programming language has its own data model, and these differ from one another, often in quite substantial ways. The basic principle under which most programming languages deal with data is that each program has access to “boxes,” which we can think of as regions of storage. Each box has a type, such as int or char. We may store in a box any value of the correct type for that box. We often refer to the values that can be stored in boxes as data objects.
SEC. 1.3
DATA MODELS
7
We may also name boxes. In general, a name for a box is any expression that denotes that box. Often, we think of the names of boxes as the variables of the program, but that is not quite right. For example, if x is a variable local to a recursive function F , then there may be many boxes named x, each associated with a different call to F . Then the true name of such a box is a combination of x and the particular call to F . Most of the data types of C are familiar: integers, floating-point numbers, characters, arrays, structures, and pointers. These are all static notions. The operations permitted on data include the usual arithmetic operations on integers and floating-point numbers, accessing operations for elements of arrays or structures, and pointer dereferencing, that is, finding the element pointed to by a pointer. These operations are part of the dynamics of the C data model. In a programming course, we would see important data models that are not part of C, such as lists, trees, and graphs. In mathematical terms, a list is a sequence of n elements, which we shall write as (a1 , a2 , . . . , an ), where a1 is the first element, a2 the second, and so on. Operations on lists include inserting new elements, deleting elements, and concatenating lists (that is, appending one list to the end of another).
Name
Dereferencing
The list data model
✦
Example 1.1. In C, a list of integers can be represented by a data structure called a linked list in which list elements are stored in cells. Lists and their cells can be defined by a type declaration such as typedef struct CELL *LIST; struct CELL { int element; struct LIST next; }; This declaration defines a self-referential structure CELL with two fields. The first is element, which holds the value of an element of the list and is of type int. The second field of each CELL is next, which holds a pointer to a cell. Note that the type LIST is really a pointer to a CELL. Thus, structures of type CELL can be linked together by their next fields to form what we usually think of as a linked list, as suggested in Fig. 1.3. The next field can be thought of as either a pointer to the next cell or as representing the entire list that follows the cell in which it appears. Similarly, the entire list can be represented by a pointer, of type LIST, to the first cell on the list. a1
•
a2
•
···
an
•
Fig. 1.3. A linked list representing the list (a1 , a2 , . . . , an ).
Cells are represented by rectangles, the left part of which is the element, and the right part of which holds a pointer, shown as an arrow to the next cell pointed to. A dot in the box holding a pointer means that the pointer is NULL.1 Lists will be covered in more detail in Chapter 6. ✦ 1
NULL is a symbolic constant defined in the standard header file stdio.h to be equal to a value that cannot be a pointer to anything. We shall use it to have this meaning throughout the book.
8
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
Data Models Versus Data Structures Despite their similar names, a “list” and a “linked list” are very different concepts. A list is a mathematical abstraction, or data model. A linked list is a data structure. In particular, it is the data structure we normally use in C and many similar languages to represent abstract lists in programs. There are other languages in which it is not necessary to use a data structure to represent abstract lists. For example, the list (a1 , a2 , . . . , an ) could be represented directly in the language Lisp and in the language Prolog similarly, as [a1 , a2 , . . . , an ].
Data Models of System Software Operating systems
Data models are found not only in programming languages but also in operating systems and applications programs. You are probably familiar with an operating system such as UNIX or MS-DOS (perhaps with Microsoft Windows).2 The function of an operating system is to manage and schedule the resources of a computer. The data model for an operating system like UNIX has concepts such as files, directories, and processes. /
mnt
usr
bin
ann
a1
a2
···
bob
a3
b1
b2
Fig. 1.4. A typical UNIX directory/file structure.
1.
The data itself is stored in files, which in the UNIX system are strings of characters.
2.
Files are organized into directories, which are collections of files and/or other directories. The directories and files form a tree with the files at the leaves.3 Figure 1.4 suggests the tree that might represent the directory structure of a typical UNIX operating system. Directories are indicated by circles. The root directory / contains directories called mnt, usr, bin, and so on. The directory /usr contains directories ann and bob; directory ann contains three files: a1, a2, and a3.
Files Directories
2
3
If you are unfamiliar with operating systems, you can skip the next paragraphs. However, most readers have probably encountered an operating system, perhaps under another name. For example, the Macintosh “system” is an operating system, although different terminology is used. For example, a directory becomes a “folder” in Macintosh-ese. However, “links” in directories may make it appear that a file or directory is part of several different directories.
SEC. 1.3
3.
Processes Pipes
✦
DATA MODELS
9
Processes are individual executions of programs. Processes take zero or more streams as input and produce zero or more streams as output. In the UNIX system, processes can be combined by pipes, where the output from one process is fed as input into the next process. The resulting composition of processes can be viewed as a single process with its own input and output.
Example 1.2. Consider the UNIX command line bc | word | speak The symbol | indicates a pipe, an operation that makes the output of the process on the left of this symbol be the input to the process on its right. The program bc is a desk calculator that takes arithmetic expressions, such as 2 + 3, as input and produces the answer 5 as output. The program word translates numbers into words; speak translates words into phoneme sequences, which are then uttered over a loudspeaker by a voice synthesizer. Connecting these three programs together by pipes turns this UNIX command line into a single process that behaves like a “talking” desk calculator. It takes as input arithmetic expressions and produces as output the spoken answers. This example also suggests that a complex task may be implemented more easily as the composition of several simpler functions. ✦
Text editors
There are many other aspects to an operating system, such as how it manages security of data and interaction with the user. However, even these few observations should make it apparent that the data model of an operating system is rather different from the data model of a programming language. Another type of data model is found in text editors. Every text editor’s data model incorporates a notion of text strings and editing operations on text. The data model usually includes the notion of lines, which, like most files, are character strings. However, unlike files, lines may have associated line numbers. Lines may also be organized into larger units such as paragraphs, and operations on lines are normally applicable anywhere within the line — not just at the front, like the most common file operations. The typical editor supports a notion of a “current” line (where the cursor is) and probably a current position within that line. Operations performed by the editor include various modifications to lines, such as deletion or insertion of characters within the line, deletion of lines, and creation of new lines. It is also possible in typical editors to search for features, such as specific character strings, among the lines of the file being edited. In fact, if you examine any other familiar piece of software, such as a spreadsheet or a video game, a pattern emerges. Each program that is designed to be used by others has its own data model, within which the user must work. The data models we meet are often radically different from one another, both in the primitives they use to represent data and in the operations on that data that are offered to the user. Yet each data model is implemented, via data structures and the programs that use them, in some programming language.
The Data Model of Circuits We shall also meet in this book a data model for computer circuits. This model, called propositional logic, is most useful in the design of computers. Computers are composed of elementary components called gates. Each gate has one or more
10
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
inputs and one output; the value of an input or output can be only 0 or 1. A gate performs a simple function — such as AND, where the output is 1 if all the inputs are 1 and the output is 0 if one or more of the inputs are 0. At one level of abstraction, computer design is the process of deciding how to connect gates to perform the basic operations of a computer. There are many other levels of abstraction associated with computer design as well. Figure 1.5 shows the usual symbol for an AND-gate, together with its truth table, which indicates the output value of the gate for each pair of input values.4 We discuss truth tables in Chapter 12 and gates and their interconnections in Chapter 13. x z
AND y
x
y
z
0 0 1 1
0 1 0 1
0 0 0 1
Fig. 1.5. An AND-gate and its truth table.
✦
Bit
Example 1.3. To execute the C assignment statement a = b+c, a computer performs the addition with an adder circuit. In the computer, all numbers are represented in binary notation using the two digits 0 and 1 (called binary digits, or bits for short). The familiar algorithm for adding decimal numbers, where we add the digits at the right end, generate a carry to the next place to the left, add that carry and the digits at that place, generate a carry to the next place to the left, and so on, works in binary as well. x
y
d
c
z Fig. 1.6. A one-bit adder: dz is the sum x + y + c.
One-bit adder
Out of a few gates, we can build a one-bit adder circuit, as suggested in Fig. 1.6. Two input bits, x and y, and a carry-in bit c, are summed, resulting in a sum bit z and a carry-out bit d. To be precise, d is 1 if two or more of c, x, and y are 1, while z is 1 if an odd number (one or three) of c, x, and y are 1, as suggested by 4
Note that if we think of 1 as “true” and 0 as “false,” then the AND-gate performs the same logical operation as the && operator of C.
SEC. 1.3
x
y
c
d
z
0 0 0 0 1 1 1 1
0 0 1 1 0 0 1 1
0 1 0 1 0 1 0 1
0 0 0 1 0 1 1 1
0 1 1 0 1 0 0 1
DATA MODELS
11
Fig. 1.7. Truth table for the one-bit adder.
The Ripple-Carry Addition Algorithm We all have used the ripple-carry algorithm to add numbers in decimal. To add 456 + 829, for example, one performs the steps suggested below: 1 4 5 6 8 2 9 5
0 4 5 6 8 2 9 8 5
4 5 6 8 2 9 1 2 8 5
That is, at the first step, we add the rightmost digits, 6+9 = 15. We write down the 5 and carry the 1 to the second column. At the second step, we add the carry-in, 1, and the two digits in the second place from the right, to get 1 + 5 + 2 = 8. We write down the 8, and the carry is 0. In the third step, we add the carry-in, 0, and the digits in the third place from the right, to get 0 + 4 + 8 = 12. We write down the 2, but since we are at the leftmost place, we do not carry the 1, but rather write it down as the leftmost digit of the answer. Binary ripple-carry addition works the same way. However, at each place, the carry and the “digits” being added are all either 0 or 1. The one-bit adder thus describes completely the addition table for a single place. That is, if all three bits are 0, then the sum is 0, and so we write down 0 and carry 0. If one of the three is 1, the sum is 1; we write down 1 and carry 0. If two of the three are 1, the sum is 2, or 10 in binary; we write down 0 and carry 1. If all three are 1, then the sum is 3, or 11 in binary, and so we write down 1 and carry 1. For example, to add 101 to 111 using binary ripple-carry addition, the steps are 1 1 0 1 1 1 1 0
1 1 0 1 1 1 1 0 0
1 0 1 1 1 1 1 1 0 0
the table of Fig. 1.7. The carry-out bit followed by the sum bit — that is, dz — forms a two-bit binary number, which is the total number of x, y, and c that are 1. In this sense, the one-bit adder adds its inputs. Many computers represent integers as 32-bit numbers. An adder circuit can
12
Ripple-carry adder
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
then be composed of 32 one-bit adders, as suggested in Fig. 1.8. This circuit is often called a ripple-carry adder, because the carry ripples from right to left, one bit at a time. Note that the carry into the rightmost (low-order bit) one-bit adder is always 0. The sequence of bits x31 x30 · · · x0 represents the bits of the first number being added, and y31 y30 · · · y0 is the second addend. The sum is dz31 z30 · · · z0 ; that is, the leading bit is the carry-out of the leftmost one-bit adder, and the following bits of the sum are the sum bits of the adders, from the left. x31
x30 y31
x0 y30
d
y0
0
···
z31
z30
z0
Fig. 1.8. A ripple-carry adder: dz31 z30 · · · z0 = x31 x30 · · · x0 + y31 y30 · · · y0 .
The circuit of Fig. 1.8 is really an algorithm in the data model of bits and the primitive operations of gates. However, it is not a particularly good algorithm. The reason is that until we compute the carry-out of the rightmost place, we cannot compute z1 or the carry-out of the second place. Until we compute the carry-out of the second place, we cannot compute z2 or the carry-out of the third place, and so on. Thus, the time taken by the circuit is the length of the numbers being added — 32 in our case — multiplied by the time needed by a one-bit adder. One might suspect that the need to “ripple” the carry through each of the onebit adders, in turn, is inherent in the definition of addition. Thus, it may come as a surprise to the reader that computers have a much faster way of adding numbers. We shall cover such an improved algorithm for addition when we discuss the design of circuits in Chapter 13. ✦
EXERCISES 1.3.1: Explain the difference between the static and dynamic aspects of a data model. 1.3.2: Describe the data model of your favorite video game. Distinguish between static and dynamic aspects of the model. Hint : The static parts are not just the parts of the game board that do not move. For example, in Pac Man, the static part includes not only the map, but the “power pills,” “monsters,” and so on. 1.3.3: Describe the data model of your favorite text editor. 1.3.4: Describe the data model of a spreadsheet program.
SEC. 1.4
✦ ✦ ✦ ✦
1.4
THE C DATA MODEL
13
The C Data Model In this section we shall highlight important parts of the data model used by the C programming language. As an example of a C program, consider the program in Fig. 1.10 that uses the variable num to count the number of characters in its input. #include <stdio.h> main() { int num; num = 0; while (getchar() != EOF) ++num; /* add 1 to num */ printf("%d\n", num); } Fig. 1.10. C program to count number of input characters.
EOF
The first line tells the C preprocessor to include as part of the source the standard input/output file stdio.h, which contains the definitions of the functions getchar and printf, and the symbolic constant EOF, a value that represents the end of a file. A C program itself consists of a sequence of definitions, which can be either function definitions or data definitions. One must be a definition of a function called main. The first statement in the function body of the program in Fig. 1.10 declares the variable num to be of type int. (All variables in a C program must be declared before their use.) The next statement initializes num to zero. The following while statement reads input characters one at a time using the library function getchar, incrementing num after each character read, until there are no more input characters. The end of file is signaled by the special value EOF on the input. The printf statement prints the value of num as a decimal integer, followed by a newline character.
The C Type System We begin with the static part of the C data model, the type system, which describes the values that data may have. We then discuss the dynamics of the C data model, that is, the operations that may be performed on data. In C, there is an infinite set of types, any of which could be the type associated with a particular variable. These types, and the rules by which they are constructed, form the type system of C. The type system contains basic types such as integers, and a collection of type-formation rules with which we can construct progressively more complex types from types we already know. The basic types of C are 1.
Characters (char, signed char, unsigned char)
2.
Integers (int, short int, long int, unsigned)
3.
Floating-point numbers (float, double, long double)
4.
Enumerations (enum)
14
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
Integers and floating-point numbers are considered to be arithmetic types. The type-formation rules assume that we already have some types, which could be basic types or other types that we have already constructed using these rules. Here are some examples of the type formation rules in C: 1.
Array types. We can form an array whose elements are type T with the declaration T A[n] This statement declares an array A of n elements, each of type T . In C, array subscripts begin at 0, so the first element is A[0] and the last element is A[n−1]. Arrays can be constructed from characters, arithmetic types, pointers, structures, unions, or other arrays.
Members of a structure
2.
Structure types. In C, a structure is a grouping of variables called members or fields. Within a structure different members can have different types, but each member must have elements of a single type. If T1 , T2 , . . . , Tn are types and M1 , M2 , . . . , Mn are member names, then the declaration struct S { T1 M1 ; T2 M2 ; ··· Tn Mn ; } defines a structure whose tag (i.e., the name of its type) is S and that has n members. The ith member has the name Mi and a value of type Ti , for i = 1, 2, . . . , n. Example 1.1 is an illustration of a structure. This structure has tag CELL and two members. The first member has name element and has integer type. The second has name next and its type is a pointer to a structure of the same type. The structure tag S is optional, but it provides a convenient shorthand for referring to the type in later declarations. For example, the declaration struct S myRecord; defines the variable myRecord to be a structure of type S.
3.
Union types. A union type allows a variable to have different types at different times during the execution of a program. The declaration union { T1 M1 ; T2 M2 ; ··· Tn Mn ; } x; defines a variable x that can hold a value of any of the types T1 , T2 , . . . , Tn . The member names M1 , M2 , . . . , Mn help indicate which type the value of x should be regarded as being. That is, x.Mi refers to the value of x treated as a value of type Ti .
SEC. 1.4
4.
THE C DATA MODEL
15
Pointer types. C is distinctive for its reliance on pointers. A variable of type pointer contains the address of a region of storage. We can access the value of another variable indirectly through a pointer. The declaration T *p; defines the variable p to be a pointer to a variable of type T . Thus p names a box of type pointer to T and the value in box p is a pointer. We often draw the value of p as an arrow, rather than as an object of type T itself, as shown in Fig. 1.11. What really appears in the box named p is the address, or location, at which an object of type T is stored in the computer. Consider the declaration int x, *p; In C, the unary operator & is used to obtain the address of an object, so the statement p = &x; assigns the address of x to p; that is, it makes p point to x. The unary operator * applied to p fetches the value of the box pointed to by p, so the statement y = *p; assigns to y the contents of whatever box p points to. If y is a variable of type int, then p = &x; y = *p; is equivalent to the assignment y = x;
p
•
Object of type T
Fig. 1.11. Variable p is of type pointer to T .
✦
Example 1.4. C has the typedef construct to create synonyms for type names. The declaration typedef int Distance;
16
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
typedef int type1[10]; typedef type1 *type2; typedef struct { int field1; type2 field2; } type3; typedef type3 type4[5]; Fig. 1.12. Some C typedef declarations.
0
field1
3
field2
•
•
9 (a)
(b)
0
(c)
field1
3
field2
•
1
···
2
···
3
···
4
field1
7
field2
• (d)
Fig. 1.13. Visualization of type declarations in Fig. 1.12.
SEC. 1.4
THE C DATA MODEL
17
Types, Names, Variables, and Identifiers A number of terms associated with data objects have different meanings but are easy to confuse. First, a type describes a “shape” for data objects. In C, a new name T may be defined for an existing type using the typedef construct typedef T Type descriptor
Here the type descriptor is an expression that tells us the shape of objects of the type T . A typedef declaration for T does not actually create any objects of that type. An object of type T is created by a declaration of the form T x; Here, x is an identifier, or “variable name.” Possibly, x is static (not local to any function), in which case the box for x is created when the program starts. If x is not static, then it is local to some function F . When F is called, a box whose name is “the x associated with this call to F ” is created. More precisely, the name of the box is x, but only uses of the identifier x during the execution of this call to F refer to this box. As mentioned in the text, there can be many boxes each of whose name involves the identifier x, since F may be recursive. There may even be other functions that also have used identifier x to name one of their variables. Moreover, names are more general than identifiers, since there are many kinds of expressions that could be used to name boxes. For instance, we mentioned that *p could be the name of an object pointed to by pointer p, and other names could be complex expressions such as (*p).f[2] or p->f[2]. The last two expressions are equivalent and refer to the array element number 2 of the field f of the structure pointed to by pointer p. allows the name Distance to be used in place of the type int. Consider the four typedef declarations in Fig. 1.12. In the conventional view of data in C, an object of type type1 is an array with 10 slots, each holding an integer, as suggested in Fig. 1.13(a). Likewise, objects of type type2 are pointers to such arrays, as in Fig. 1.13(b). Structures, like those of type3, are visualized as in Fig. 1.13(c), with a slot for each field; note that the name of the field (e.g., field1) does not actually appear with the value of the field. Finally, objects of the array type type4 would have five slots, each of which holds an object of type type3, a structure we suggest in Fig. 1.13(d). ✦
Functions
Return-value
Functions also have associated types, even though we do not associate boxes or “values” with functions, as we do with program variables. For any list of types T1 , T2 , . . . , Tn , we can define a function with n parameters consisting of those types, in order. This list of types followed by the type of the value returned by the function (the return-value) is the “type” of the function. If the function has no return value, its type is void. In general, we can build types by applying the type-construction rules arbitrarily, but there are a number of constraints. For example, we cannot construct an
18
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
“array of functions,” although we can construct an array of pointers to functions. The complete set of rules for constructing types in C can be found in the ANSI standard.
Operations in the C Data Model The operations on data in the C data model can be divided into three categories: 1.
Operations that create or destroy a data object.
2.
Operations that access and modify parts of a data object.
3.
Operations that combine the values of data objects to form a new value for a data object.
Data Object Creation and Disposal For data creation, C provides several rudimentary mechanisms. When a function is called, boxes for each of its local arguments (parameters) are created; these serve to hold the values of the arguments. Another mechanism for data creation is through the use of the library routine malloc(n), which returns a pointer to n consecutive character positions of unused storage that can be used to store data by the caller of malloc. Data objects can then be created in this storage region. C has the analogous methods for destroying data objects. Local parameters of a function call cease to exist when the function returns. The routine free releases the storage created by malloc. In particular, the effect of calling free(p) is to release the storage area pointed to by p. It is disastrous to use free to get rid of an object that was not created by calling malloc.
Data Access and Modification C has mechanisms for accessing the components of objects. It uses a[i] to access the ith element of array a, x.m to access member m of a structure named x, and *p to access the object pointed to by pointer p. Modifying, or writing, values in C is done principally by the assignment operators, which allow us to change the value of an object.
✦
Example 1.5. If a is a variable of type type4 defined in Example 1.4, then a[0].(*field2[3]) = 99; assigns the value 99 to the fourth element of the array pointed to by field2 in the structure that is the first element of the array a. ✦
Data Combination C has a rich set of operators for manipulating and combining values. The principal operators are 1.
Arithmetic operators. C provides:
SEC. 1.4
2.
THE C DATA MODEL
19
a)
The customary binary arithmetic operators +, −, ∗, / on integers and floating-point numbers. Integer division truncates (4/3 yields 1).
b)
There are the unary + and − operators.
c)
The modulus operator i % j produces the remainder when i is divided by j.
d)
The increment and decrement operators, ++ and --, applied to a single integer variable add or subtract 1 from that variable, respectively. These operators can appear before or after their operand, depending on whether we wish the value of the expression to be computed before or after the change in the variable’s value.
Logical operators. C does not have a Boolean type; it uses zero to represent the logical value false, and nonzero to represent true.5 C uses: a)
&& to represent AND. For example, the expression x && y returns 1 if both operands are nonzero, 0 otherwise. However, y is not evaluated if x has the value 0.
b)
|| represents OR. The expression x || y returns 1 if either x or y is nonzero, and returns 0 otherwise. However, y is not evaluated if x is nonzero.
c)
The unary negation operator !x returns 0 if x is nonzero and returns 1 if x = 0.
d)
The conditional operator is a ternary (3-argument) operator represented by a question mark and a colon. The expression x?y:z returns the value of y if x is true (i.e., it is nonzero) and returns the value of z if x is false (i.e., 0).
3.
Comparison operators. The result of applying one of the six relational comparison operators (==, !=, <, >, <=, and >=) to integers or floating point numbers is 0 if the relation is false and 1 otherwise.
4.
Bitwise manipulation operators. C provides several useful bitwise logical operators, which treat integers as if they were bits strings equal to their binary representations. These include & for bitwise AND, | for bitwise inclusive-or, ^ for bitwise exclusive-or, << for left shift, >> for right shift, and a tilde for left shift.
5.
Assignment operators. C uses = as the assignment operator. In addition, C allows expressions such as x = x + y; to be written in a shortened form x += y; Similar forms apply to the other binary arithmetic operators.
5
We shall use TRUE and FALSE as defined constants 1 and 0, respectively, to represent Boolean values; see Section 1.6.
20
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
6.
Coercion operators. Coercion is the process of converting a value of one type into an equivalent value of another type. For example, if x is a floating-point number and i is an integer, then x = i causes the integer value of i to be converted to a floating-point number with the same value. Here, the coercion operator is not shown explicitly, but the C compiler can deduce that conversion from integer to float is necessary and inserts the required step.
EXERCISES 1.4.1: Explain the difference between an identifier of a C program and a name (for a “box” or data object). 1.4.2: Give an example of a C data object that has more than one name. 1.4.3: If you are familiar with another programming language besides C, describe its type system and operations. ✦ ✦ ✦ ✦
1.5
Algorithms and the Design of Programs The study of data models, their properties, and their appropriate use is one pillar of computer science. A second, equally important pillar is the study of algorithms and their associated data structures. We need to know the best ways to perform common tasks, and we need to learn the principal techniques for designing good algorithms. Further, we need to understand how the use of data structures and algorithms fits into the process of creating useful programs. The themes of data models, algorithms, data structures, and their implementation in programs are interdependent, and each appears many times throughout the book. In this section, we shall mention some generalities regarding the design and implementation of programs.
The Creation of Software
Software development process
In a programming class, when you were given a programming problem, you probably designed an algorithm to solve the problem, implemented the algorithm in some language, compiled and ran the program on some sample data, and then submitted the program to be graded. In a commercial setting, programs are written under rather different circumstances. Algorithms, at least those simple enough and common enough to have names, are usually small parts of a complete program. Programs, in turn, are usually components of a larger system, involving hardware as well as software. Both the programs and the complete systems in which they are embedded are developed by teams of programmers and engineers; there could be hundreds of people on such a team. The development of a software system typically spans several phases. Although these phases may superficially bear some resemblance to the steps involved in solving the classroom programming assignment, most of the effort in building a software system to solve a given problem is not concerned with programming. Here is an idealized scenario.
SEC. 1.5
Prototyping
Software reuse
ALGORITHMS AND THE DESIGN OF PROGRAMS
21
Problem definition and specification. The hardest, but most important, part of the task of creating a software system is defining what the problem really is and then specifying what is needed to solve it. Usually, problem definition begins by analyzing the users’ requirements, but these requirements are often imprecise and hard to write down. The system architect may have to consult with the future users of the system and iterate the specification, until both the specifier and the users are satisfied that the specification defines and solves the problem at hand. In the specification stage, it may be helpful to build a simple prototype or model of the final system, to gain insight into its behavior and intended use. Data modeling is also an important tool in the problem-definition phase. Design. Once the specification is complete, a high-level design of the system is created, with the major components identified. A document outlining the high-level design is prepared, and performance requirements for the system may be included. More detailed specifications of some of the major components may also be included during this phase. A cost-effective design often calls for the reuse or modification of previously constructed components. Various software methodologies such as objectoriented technology facilitate the reuse of components. Implementation. Once the design is fixed, implementation of the components can proceed. Many of the algorithms discussed in this book are useful in the implementation of new components. Once a component has been implemented, it is subject to a series of tests to make sure that it behaves as specified. Integration and system testing. When the components have been implemented and individually tested, the entire system is assembled and tested. Installation and field testing. Once the developer is satisfied that the system will work to the customer’s satisfaction, the system is installed on the customer’s premises and the final field testing takes place. Maintenance. At this point, we might think that the bulk of the work has been done. Maintenance remains, however, and in many situations maintenance can account for more than half the cost of system development. Maintenance may involve modifying components to eliminate unforeseen side-effects, to correct or improve system performance, or to add features. Because maintenance is such an important part of software systems design, it is important to write programs that are correct, rugged, efficient, modifiable, and — whenever possible — portable from one computer to another. It is very important to catch errors as early as possible, preferably during the problem-definition phase. At each successive phase, the cost of fixing a design error or programming bug rises greatly. Independent reviews of requirements and designs are beneficial in reducing downstream errors.
Programming Style An individual programmer can ease the maintenance burden greatly by writing programs that others can read and modify readily. Good programming style comes only with practice, and we recommend that you begin at once to try writing programs that are easy for others to understand. There is no magic formula that will guarantee readable programs, but there are several useful rules of thumb:
22
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
1.
Modularize a program into coherent pieces.
2.
Lay out a program so that its structure is clear.
3.
Write intelligent comments to explain a program. Describe, clearly and precisely, the underlying data models, the data structures selected to represent them, and the operation performed by each procedure. When describing a procedure, state the assumptions made about its inputs, and tell how the output relates to the input.
4.
Use meaningful names for procedures and variables.
Defined constants
5.
Avoid explicit constants whenever possible. For example, do not use 7 for the number of dwarfs. Rather, use a defined constant such as NumberOfDwarfs, so that you can easily change all uses of this constant to 8, if you decide to add another dwarf.
Global variables
6.
Avoid the use of “global variables” — that is, variables defined for the program as a whole — unless the data represented by that variable really is used by most of the procedures of the program.
Test suite
Another good programming practice is to maintain a test suite of inputs that will try to exercise every line of code while you are developing a program. Whenever new features are added to the program, the test suite can be run to make sure that the new program has the same behavior as the old on previously working inputs.
✦ ✦ ✦ ✦
1.6
Some C Conventions Used Throughout the Book There are several definitions and conventions that we shall find useful as we illustrate concepts with C programs. Some of these are common conventions found in the standard header file stdio.h, while others are defined specially for the purposes of this book and must be included with any C program that uses them.
NULL
1.
The identifier NULL is a value that may appear anywhere a pointer can appear, but it is not a value that can ever point to anything. Thus, NULL in a field such as next in the cells of Example 1.1 can be used to indicate the end of a list. We shall see that NULL has a number of similar uses in other data structures. NULL is properly defined in stdio.h.
2.
The identifiers TRUE and FALSE are defined by #define TRUE 1 #define FALSE 0 Thus, TRUE can be used anywhere a condition with logical value true is wanted, and FALSE can be used for a condition whose value is false.
TRUE and FALSE BOOLEAN
3.
The type BOOLEAN is defined as typedef int BOOLEAN; We use BOOLEAN whenever we want to stress the fact that we are interested in the logical rather than the numeric value of an expression.
SEC. 1.7
EOF
SUMMARY OF CHAPTER 1
23
4.
The identifier EOF is a value that is returned by file-reading functions such as getchar() when there are no more bytes left to be read from the file. An appropriate value for EOF is provided in stdio.h.
5.
We shall define a macro that generates declarations of cells of the kind used in Example 1.1. An appropriate definition appears in Fig. 1.14. It declares cells with two fields: an element field whose type is given by the parameter Type and a next field to point to a cell with this structure. The macro provides two external definitions: CellName is the name of structures of this type, and ListName is a name for the type of pointers to these cells.
Cell definition
#define DefCell(EltType, CellType, ListType) typedef struct CellType *ListType; struct CellType { EltType element; ListType next; }
\ \ \ \ \
Fig. 1.14. A macro for defining list cells.
✦
Example 1.6. We can define cells of the type used in Example 1.1 by the macro use DefCell(int, CELL, LIST); The macro then expands into typedef struct CELL, LIST; struct CELL { int element; LIST next; } As a consequence, we can use CELL as the type of integer cells, and we can use LIST as the type of pointers to these cells. For example, CELL c; LIST L; defines c to be a cell and L to be a pointer to a cell. Note that the representation of a list of cells is normally a pointer to the first cell on the list, or NULL if the list is empty. ✦
✦ ✦ ✦ ✦
1.7
Summary of Chapter 1 At this point you should be aware of the following concepts: ✦
How data models, data structures, and algorithms are used to solve problems
✦ ✦ ✦ ✦
1.8
24
COMPUTER SCIENCE: THE MECHANIZATION OF ABSTRACTION
✦
The distinction between a list as a data model and a linked list as a data structure
✦
The presence of some kind of data model in every software system, be it a programming language, an operating system, or an application program
✦
The key elements of the data model supported by the programming language C
✦
The major steps in the development of a large software system
Bibliographic Notes for Chapter 1 Kernighan and Ritchie [1988] is the classic reference for the C programming language. Roberts [1994] is a good introduction to programming using C. Stroustrup [1991] has created an object-oriented extension of C called C++ that is now widely used for implementing systems. Sethi [1989] provides an introduction to the data models of several major programming languages. Brooks [1974] eloquently describes the technical and managerial difficulties in developing large software systems. Kernighan and Plauger [1978] provide sound advice for improving your programming style. American National Standards Institute (ANSI) [1990]. Programming Language C, American National Standards Institute, New York. Brooks, F. P. [1974]. The Mythical Man Month, Addison-Wesley, Reading, Mass. Kernighan, B. W., and P. J. Plauger [1978]. The Elements of Programming Style, second edition, McGraw-Hill, New York. Kernighan, B. W., and D. M. Ritchie [1988]. The C Programming Language, second edition, Prentice-Hall, Englewood Cliffs, New Jersey. Roberts, E. S. [1994]. A C-Based Introduction to Computer Science, AddisonWesley, Reading, Mass. Sethi, R. [1989]. Programming Languages: Concepts and Constructs, AddisonWesley, Reading, Mass. Stroustrup, B. [1991]. The C++ Programming Language, second edition, AddisonWesley, Reading, Mass.
2
CHAPTER
Iteration, Induction, and Recursion
✦ ✦ ✦ ✦
The power of computers comes from their ability to execute the same task, or different versions of the same task, repeatedly. In computing, the theme of iteration is met in a number of guises. Many concepts in data models, such as lists, are forms of repetition, as “A list either is empty or is one element followed by another, then another, and so on.” Programs and algorithms use iteration to perform repetitive jobs without requiring a large number of similar steps to be specified individually, as “Do the next step 1000 times.” Programming languages use looping constructs, like the while- and for-statements of C, to implement iterative algorithms. Closely related to repetition is recursion, a technique in which a concept is defined, directly or indirectly, in terms of itself. For example, we could have defined a list by saying “A list either is empty or is an element followed by a list.” Recursion is supported by many programming languages. In C, a function F can call itself, either directly from within the body of F itself, or indirectly by calling some other function, which calls another, and another, and so on, until finally some function in the sequence calls F . Another important idea, induction, is closely related to “recursion” and is used in many mathematical proofs. Iteration, induction, and recursion are fundamental concepts that appear in many forms in data models, data structures, and algorithms. The following list gives some examples of uses of these concepts; each will be covered in some detail in this book. 1.
Iterative techniques. The simplest way to perform a sequence of operations repeatedly is to use an iterative construct such as the for-statement of C.
2.
Recursive programming. C and many other languages permit recursive functions, which call themselves either directly or indirectly. Often, beginning programmers are more secure writing iterative programs than recursive ones, but an important goal of this book is to accustom the reader to thinking and programming recursively, when appropriate. Recursive programs can be simpler to write, analyze, and understand. 25
26
ITERATION, INDUCTION, AND RECURSION
Notation: The Summation and Product Symbols AnP oversized Greek capital letter sigma is often used to denote a summation, as in ni=1 i. This particular expression represents the sum of the integers from 1 to n; that is, it stands for the sum 1 + 2 + 3 + · · · + n. More generally, we can sum any function f (i) of the summation index i. (Of course, the index could be some Pb symbol other than i.) The expression i=a f (i) stands for f (a) + f (a + 1) + f (a + 2) + · · · + f (b) Pm For example, j=2 j 2 stands for the sum 4 + 9 + 16 + · · · + m2 . Here, the function f is “squaring,” and we used index j instead of i. Pb As a special case, if b < a, then there are no terms in the sum i=a f (i), and the value of the expression, by convention, is taken to be 0. P If b = a, then there is a exactly one term, that for i = a. Thus, the value of the sum i=a f (i) is just f (a). The Qb analogous notation for products uses an oversized capital pi. The expression i=a f (i) stands for the product f (a) × f (a + 1) × f (a + 2) × · · · × f (b); if b < a, the product is taken to be 1.
3.
Basis Inductive step
Proofs by induction. An important technique for showing that a statement is true is “proof by induction.” We shall cover inductive proofs extensively, starting in Section 2.3. The following is the simplest form of an inductive proof. We begin with a statement S(n) involving a variable n; we wish to prove that S(n) is true. We prove S(n) by first proving a basis, that is, the statement S(n) for a particular value of n. For example, we could let n = 0 and prove the statement S(0). Second, we must prove an inductive step, in which we prove that the statement S, for one value of its argument, follows from the same statement S for the previous values of its argument; that is, S(n) implies S(n + 1) for all n ≥ 0. For example, S(n) might be the familiar summation formula n X i = n(n + 1)/2 (2.1) i=1
which says that the sum of the integers from 1 to n equals n(n+1)/2. The basis could be S(1) — that is, Equation (2.1) with 1 in place of nP— which is just the equality 1 = 1 × 2/2. The inductive step is to show that ni=1 i = n(n + 1)/2 Pn+1 implies that i=1 i = (n + 1)(n + 2)/2; the former is S(n), which is Equation (2.1) itself, while the latter is S(n + 1), which is Equation (2.1) with n + 1 replacing n everywhere n appears. Section 2.3 will show us how to construct proofs such as this. 4.
Proofs of program correctness. In computer science, we often wish to prove, formally or informally, that a statement S(n) about a program is true. The statement S(n) might, for example, describe what is true on the nth iteration of some loop or what is true for the nth recursive call to some function. Proofs of this sort are generally inductive proofs.
5.
Inductive definitions. Many important concepts of computer science, especially those involving data models, are best defined by an induction in which we give
SEC. 2.2
ITERATION
27
a basis rule defining the simplest example or examples of the concept, and an inductive rule or rules, where we build larger instances of the concept from smaller ones. For instance, we noted that a list can be defined by a basis rule (an empty list is a list) together with an inductive rule (an element followed by a list is also a list). 6.
Analysis of running time. An important criterion for the “goodness” of an algorithm is how long it takes to run on inputs of various sizes (its “running time”). When the algorithm involves recursion, we use a formula called a recurrence equation, which is an inductive definition that predicts how long the algorithm takes to run on inputs of different sizes.
Each of these subjects, except the last, is introduced in this chapter; the running time of a program is the topic of Chapter 3. ✦ ✦ ✦ ✦
2.1
What This Chapter Is About In this chapter we meet the following major concepts. ✦
Iterative programming (Section 2.2)
✦
Inductive proofs (Sections 2.3 and 2.4)
✦
Inductive definitions (Section 2.6)
✦
Recursive programming (Sections 2.7 and 2.8)
✦
Proving the correctness of a program (Sections 2.5 and 2.9)
In addition, we spotlight, through examples of these concepts, several interesting and important ideas from computer science. Among these are
✦ ✦ ✦ ✦
2.2
✦
Sorting algorithms, including selection sort (Section 2.2) and merge sort (Section 2.8)
✦
Parity checking and detection of errors in data (Section 2.3)
✦
Arithmetic expressions and their transformation using algebraic laws (Sections 2.4 and 2.6)
✦
Balanced parentheses (Section 2.6)
Iteration Each beginning programmer learns to use iteration, employing some kind of looping construct such as the for- or while-statement of C. In this section, we present an example of an iterative algorithm, called “selection sort.” In Section 2.5 we shall prove by induction that this algorithm does indeed sort, and we shall analyze its running time in Section 3.6. In Section 2.8, we shall show how recursion can help us devise a more efficient sorting algorithm using a technique called “divide and conquer.”
28
ITERATION, INDUCTION, AND RECURSION
Common Themes: Self-Definition and Basis-Induction As you study this chapter, you should be alert to two themes that run through the various concepts. The first is self-definition, in which a concept is defined, or built, in terms of itself. For example, we mentioned that a list can be defined as being empty or as being an element followed by a list. The second theme is basis-induction. Recursive functions usually have some sort of test for a “basis” case where no recursive calls are made and an “inductive” case where one or more recursive calls are made. Inductive proofs are well known to consist of a basis and an inductive step, as do inductive definitions. This basisinduction pairing is so important that these words are highlighted in the text to introduce each occurrence of a basis case or an inductive step. There is no paradox or circularity involved in properly used self-definition, because the self-defined subparts are always “smaller” than the object being defined. Further, after a finite number of steps to smaller parts, we arrive at the basis case, at which the self-definition ends. For example, a list L is built from an element and a list that is one element shorter than L. When we reach a list with zero elements, we have the basis case of the definition of a list: “The empty list is a list.” As another example, if a recursive function works, the arguments of the call must, in some sense, be “smaller” than the arguments of the calling copy of the function. Moreover, after some number of recursive calls, we must get to arguments that are so “small” that the function does not make any more recursive calls.
Sorting To sort a list of n elements we need to permute the elements of the list so that they appear in nondecreasing order.
✦
Example 2.1. Suppose we are given the list of integers (3, 1, 4, 1, 5, 9, 2, 6, 5). We sort this list by permuting it into the sequence (1, 1, 2, 3, 4, 5, 5, 6, 9). Note that sorting not only orders the values so that each is either less than or equal to the one that follows, but it also preserves the number of occurrences of each value. Thus, the sorted list has two 1’s, two 5’s, and one each of the numbers that appear once in the original list. ✦
Sorted list
We can sort a list of elements of any type as long as the elements have a “lessthan” order defined on them, which we usually represent by the symbol <. For example, if the values are real numbers or integers, then the symbol < stands for the usual less-than relation on reals or integers, and if the values are character strings, we would use the lexicographic order on strings. (See the box on “Lexicographic Order.”) Sometimes when the elements are complex, such as structures, we might use only a part of each element, such as one particular field, for the comparison. The comparison a ≤ b means, as always, that either a < b or a and b are the same value. A list (a1 , a2 , . . . , an ) is said to be sorted if a1 ≤ a2 ≤ · · · ≤ an ; that is, if the values are in nondecreasing order. Sorting is the operation of taking an arbitrary list (a1 , a2 , . . . , an ) and producing a list (b1 , b2 , . . . , bn ) such that
SEC. 2.2
ITERATION
29
Lexicographic Order The usual way in which two character strings are compared is according to their lexicographic order. Let c1 c2 · · · ck and d1 d2 · · · dm be two strings, where each of the c’s and d’s represents a single character. The lengths of the strings, k and m, need not be the same. We assume that there is a < ordering on characters; for example, in C characters are small integers, so character constants and variables can be used as integers in arithmetic expressions. Thus we can use the conventional < relation on integers to tell which of two characters is “less than” the other. This ordering includes the natural notion that lower-case letters appearing earlier in the alphabet are “less than” lower-case letters appearing later in the alphabet, and the same holds for upper-case letters. We may then define the ordering on character strings called the lexicographic, dictionary, or alphabetic ordering, as follows. We say c1 c2 · · · ck < d1 d2 · · · dm if either of the following holds: Proper prefix
1.
The first string is a proper prefix of the second, which means that k < m and for i = 1, 2, . . . , k we have ci = di . According to this rule, bat < batter. As a special case of this rule, we could have k = 0, in which case the first string has no characters in it. We shall use ǫ, the Greek letter epsilon, to denote the empty string, the string with zero characters. When k = 0, rule (1) says that ǫ < s for any nonempty string s.
2.
For some value of i > 0, the first i − 1 characters of the two strings agree, but the ith character of the first string is less than the ith character of the second string. That is, cj = dj for j = 1, 2, . . . , i − 1, and ci < di . According to this rule, ball < base, because the two words first differ at position 3, and at that position ball has an l, which precedes the character s found in the third position of base.
1.
List (b1 , b2 , . . . , bn ) is sorted.
2.
List (b1 , b2 , . . . , bn ) is a permutation of the original list. That is, each value appears in list (a1 , a2 , . . . , an ) exactly as many times as that value appears in list (b1 , b2 , . . . , bn ).
Empty string
Permutation
A sorting algorithm takes as input an arbitrary list and produces as output a sorted list that is a permutation of the input.
✦
Example 2.2. Consider the list of words base, ball, mound, bat, glove, batter Given this input, and using lexicographic order, a sorting algorithm would produce this output: ball, base, bat, batter, glove, mound. ✦
Selection Sort: An Iterative Sorting Algorithm Suppose we have an array A of n integers that we wish to sort into nondecreasing
30
ITERATION, INDUCTION, AND RECURSION
Convention Regarding Names and Values We can think of a variable as a box with a name and a value. When we refer to a variable, such as abc, we use the constant-width, or “computer” font for its name, as we did in this sentence. When we refer to the value of the variable abc, we shall use italics, as abc. To summarize, abc refers to the name of the box, and abc to its contents. order. We may do so by iterating a step in which a smallest element1 not yet part of the sorted portion of the array is found and exchanged with the element in the first position of the unsorted part of the array. In the first iteration, we find (“select”) a smallest element among the values found in the full array A[0..n-1] and exchange it with A[0].2 In the second iteration, we find a smallest element in A[1..n-1] and exchange it with A[1]. We continue these iterations. At the start of the i + 1st iteration, A[0..i-1] contains the i smallest elements in A sorted in nondecreasing order, and the remaining elements of the array are in no particular order. A picture of A just before the i + 1st iteration is shown in Fig. 2.1. sorted ↑ 0
unsorted ↑ i
↑ n−1
Fig. 2.1. Picture of array just before the i + 1st iteration of selection sort.
In the i + 1st iteration, we find a smallest element in A[i..n-1] and exchange it with A[i]. Thus, after the i + 1st iteration, A[0..i] contains the i + 1 smallest elements sorted in nondecreasing order. After the (n − 1)st iteration, the entire array is sorted. A C function for selection sort is shown in Fig. 2.2. This function, whose name is SelectionSort, takes an array A as the first argument. The second argument, n, is the length of array A. Lines (2) through (5) select a smallest element in the unsorted part of the array, A[i..n-1]. We begin by setting the value of index small to i in line (2). The forloop of lines (3) through (5) consider all higher indexes j in turn, and small is set to j if A[j] has a smaller value than any of the array elements in the range A[i..j-1]. As a result, we set the variable small to the index of the first occurrence of the smallest element in A[i..n-1]. After choosing a value for the index small, we exchange the element in that position with the element in A[i], in lines (6) to (8). If small = i, the exchange is performed, but has no effect on the array. Notice that in order to swap two elements, we need a temporary place to store one of them. Thus, we move the value 1
We say “a smallest element” rather than “the smallest element” because there may be several occurrences of the smallest value. If so, we shall be happy with any of those occurrences.
2
To describe a range of elements within an array, we adopt a convention from the language Pascal. If A is an array, then A[i..j] denotes those elements of A with indexes from i to j, inclusive.
SEC. 2.2
(1)
(2) (3) (4) (5)
(6) (7) (8)
ITERATION
31
void SelectionSort(int A[], int n) { int i, j, small, temp; for (i = 0; i < n-1; i++) { /* set small to the index of the first occur- */ /* rence of the smallest element remaining */ small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j; /* when we reach here, small is the index of */ /* the first smallest element in A[i..n-1]; */ /* we now exchange A[small] with A[i] */ temp = A[small]; A[small] = A[i]; A[i] = temp; } } Fig. 2.2. Iterative selection sort.
in A[small] to temp at line (6), move the value in A[i] to A[small] at line (7), and finally move the value originally in A[small] from temp to A[i] at line (8).
✦
Example 2.3. Let us study the behavior of SelectionSort on various inputs. First, let us look at what happens when we run SelectionSort on an array with no elements. When n = 0, the body of the for-loop of line (1) is not executed, so SelectionSort does “nothing” gracefully. Now let us consider the case in which the array has only one element. Again, the body of the for-loop of line (1) is not executed. That response is satisfactory, because an array consisting of a single element is always sorted. The cases in which n is 0 or 1 are important boundary conditions, on which it is important to check the performance of any algorithm or program. Finally, let us run SelectionSort on a small array with four elements, where A[0] through A[3] are A
0 1 2 3 40 30 20 10
We begin the outer loop with i = 0, and at line (2) we set small to 0. Lines (3) to (5) form an inner loop, in which j is set to 1, 2, and 3, in turn. With j = 1, the test of line (4) succeeds, since A[1], which is 30, is less than A[small], which is A[0], or 40. Thus, we set small to 1 at line (5). At the second iteration of lines (3) to (5), with j = 2, the test of line (4) again succeeds, since A[2] < A[1], and so we set small to 2 at line (5). At the last iteration of lines (3) to (5), with j = 3, the test of line (4) succeeds, since A[3] < A[2], and we set small to 3 at line (5). We now fall out of the inner loop to line (6). We set temp to 10, which is A[small], then A[3] to A[0], or 40, at line (7), and A[0] to 10 at line (8). Now, the
32
ITERATION, INDUCTION, AND RECURSION
Sorting on Keys When we sort, we apply a comparison operation to the values being sorted. Often the comparison is made only on specific parts of the values and the part used in the comparison is called the key. For example, a course roster might be an array A of C structures of the form struct STUDENT { int studentID; char *name; char grade; } A[MAX]; We might want to sort by student ID, or name, or grade; each in turn would be the key. For example, if we wish to sort structures by student ID, we would use the comparison A[j].studentID < A[small].studentID at line (4) of SelectionSort. The type of array A and temporary temp used in the swap would be struct STUDENT, rather than integer. Note that entire structures are swapped, not just the key fields. Since it is time-consuming to swap whole structures, a more efficient approach is to use a second array of pointers to STUDENT structures and sort only the pointers in the second array. The structures themselves remain stationary in the first array. We leave this version of selection sort as an exercise.
first iteration of the outer loop is complete, and array A appears as A
0 1 2 3 10 30 20 40
The second iteration of the outer loop, with i = 1, sets small to 1 at line (2). The inner loop sets j to 2 initially, and since A[2] < A[1], line (5) sets small to 2. With j = 3, the test of line (4) fails, since A[3] ≥ A[2]. Hence, small = 2 when we reach line (6). Lines (6) to (8) swap A[1] with A[2], leaving the array A
0 1 2 3 10 20 30 40
Although the array now happens to be sorted, we still iterate the outer loop once more, with i = 2. We set small to 2 at line (2), and the inner loop is executed only with j = 3. Since the test of line (4) fails, small remains 2, and at lines (6) through (8), we “swap” A[2] with itself. The reader should check that the swapping has no effect when small = i. ✦ Figure 2.3 shows how the function SelectionSort can be used in a complete program to sort a sequence of n integers, provided that n ≤ 100. Line (1) reads and stores n integers in an array A. If the number of inputs exceeds MAX, only the first MAX integers are put into A. A message warning the user that the number of inputs is too large would be useful here, but we omit it.
SEC. 2.2
ITERATION
33
Line (3) calls SelectionSort to sort the array. Lines (4) and (5) print the integers in sorted order. #include <stdio.h> #define MAX 100 int A[MAX]; void SelectionSort(int A[], int n);
(1) (2) (3) (4) (5)
main() { int i, n; /* read and store input in A */ for (n = 0; n < MAX && scanf("%d", &A[n]) != EOF; n++) ; SelectionSort(A,n); /* sort A */ for (i = 0; i < n; i++) printf("%d\n", A[i]); /* print A */ } void SelectionSort(int A[], int n) { int i, j, small, temp; for (i = 0; i < n-1; i++) { small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j; temp = A[small]; A[small] = A[i]; A[i] = temp; } }
Fig. 2.3. A sorting program using selection sort.
EXERCISES 2.2.1: Simulate the function SelectionSort on an array containing the elements a) b) c)
6, 8, 14, 17, 23 17, 23, 14, 6, 8 23, 17, 14, 8, 6
How many comparisons and swaps of elements are made in each case? 2.2.2**: What are the minimum and maximum number of (a) comparisons and (b) swaps that SelectionSort can make in sorting a sequence of n elements?
34
ITERATION, INDUCTION, AND RECURSION
2.2.3: Write a C function that takes two linked lists of characters as arguments and returns TRUE if the first string precedes the second in lexicographic order. Hint : Implement the algorithm for comparing character strings that was described in this section. Use recursion by having the function call itself on the tails of the character strings when it finds that the first characters of both strings are the same. Alternatively, one can develop an iterative algorithm to do the same. 2.2.4*: Modify your program from Exercise 2.2.3 to ignore the case of letters in comparisons. 2.2.5: What does selection sort do if all elements are the same? 2.2.6: Modify Fig. 2.3 to perform selection sort when array elements are not integers, but rather structures of type struct STUDENT, as defined in the box “Sorting on Keys.” Suppose that the key field is studentID. 2.2.7*: Further modify Fig. 2.3 so that it sorts elements of an arbitrary type T . You may assume, however, that there is a function key that takes an element of type T as argument and returns the key for that element, of some arbitrary type K. Also assume that there is a function lt that takes two elements of type K as arguments and returns TRUE if the first is “less than” the second, and FALSE otherwise. 2.2.8: Instead of using integer indexes into the array A, we could use pointers to integers to indicate positions in the array. Rewrite the selection sort algorithm of Fig. 2.3 using pointers. 2.2.9*: As mentioned in the box on “Sorting on Keys,” if the elements to be sorted are large structures such as type STUDENT, it makes sense to leave them stationary in an array and sort pointers to these structures, found in a second array. Write this variation of selection sort. 2.2.10: Write an iterative program to print the distinct elements of an integer array. P Q 2.2.11: Use the and notations described at the beginning of this chapter to express the following. a)
The sum of the odd integers from 1 to 377
b)
The sum of the squares of the even integers from 2 to n (assume that n is even)
c)
The product of the powers of 2 from 8 to 2k
2.2.12: Show that when small = i, lines (6) through (8) of Fig. 2.2 (the swapping steps) do not have any effect on array A. ✦ ✦ ✦ ✦
2.3
Inductive Proofs Mathematical induction is a useful technique for proving that a statement S(n) is true for all nonnegative integers n, or, more generally, for all integers at or above some lower limit. For P example, in the introduction to this chapter we suggested that the statement ni=1 i = n(n + 1)/2 can be proved true for all n ≥ 1 by an induction on n. Now, let S(n) be some arbitrary statement about an integer n. In the simplest form of an inductive proof of the statement S(n), we prove two facts:
SEC. 2.3
INDUCTIVE PROOFS
35
Naming the Induction Parameter It is often useful to explain an induction by giving the intuitive meaning of the variable n in the statement S(n) that we are proving. If n has no special meaning, as in Example 2.4, we simply say “The proof is by induction on n.” In other cases, n may have a physical meaning, as in Example 2.6, where n is the number of bits in the code words. There we can say, “The proof is by induction on the number of bits in the code words.” 1.
The basis case, which is frequently taken to be S(0). However, the basis can be S(k) for any integer k, with the understanding that then the statement S(n) is proved only for n ≥ k.
2.
The inductive step, where we prove that for all n ≥ 0 [or for all n ≥ k, if the basis is S(k)], S(n) implies S(n + 1). In this part of the proof, we assume that the statement S(n) is true. S(n) is called the inductive hypothesis, and assuming it to be true, we must then prove that S(n + 1) is true.
Inductive hypothesis
S(n)
···
S(2)
S(1)
S(0)
Fig. 2.4. In an inductive proof, each instance of the statement S(n) is proved using the statement for the next lower value of n.
Figure 2.4 illustrates an induction starting at 0. For each integer n, there is a statement S(n) to prove. The proof for S(1) uses S(0), the proof for S(2) uses S(1), and so on, as represented by the arrows. The way each statement depends on the previous one is uniform. That is, by one proof of the inductive step, we prove each of the steps implied by the arrows in Fig. 2.4.
✦
Example 2.4. As an example of mathematical induction, let us prove STATEMENT
S(n):
n X
2i = 2n+1 − 1 for any n ≥ 0.
i=0
That is, the sum of the powers of 2, from the 0th power to the nth power, is 1 less than the (n + 1)st power of 2.3 For example, 1 + 2 + 4 + 8 = 16 − 1. The proof proceeds as follows. BASIS. To prove the basis, we substitute 0 for n in the equation S(n). Then S(n) becomes 3
S(n) can be proved without induction, using the formula for the sum of a geometric series. However, it will serve as a simple example of the technique of mathematical induction. Further, the proofs of the formulas for the sum of a geometric or arithmetic series that you have probably seen in high school are rather informal, and strictly speaking, mathematical induction should be used to prove those formulas.
36
ITERATION, INDUCTION, AND RECURSION
0 X
2i = 21 − 1
(2.2)
i=0
There is only one term, for i = 0, in the summation on the left side of Equation (2.2), so that the left side of (2.2) sums to 20 , or 1. The right side of Equation (2.2), which is 21 − 1, or 2 − 1, also has value 1. Thus we have proved the basis of S(n); that is, we have shown that this equality is true for n = 0. INDUCTION. Now we must prove the inductive step. We assume that S(n) is true, and we prove the same equality with n + 1 substituted for n. The equation to be proved, S(n + 1), is n+1 X
2i = 2n+2 − 1
(2.3)
i=0
To prove Equation (2.3), we begin by considering the sum on the left side, n+1 X
2i
i=0
This sum is almost the same as the sum on the left side of S(n), which is n X
2i
i=0
except that (2.3) also has a term for i = n + 1, that is, the term 2n+1 . Since we are allowed to assume that the inductive hypothesis S(n) is true in our proof of Equation (2.3), we should contrive to use S(n) to advantage. We do so by breaking the sum in (2.3) into two parts, one of which is the sum in S(n). That is, we separate out the last term, where i = n + 1, and write n+1 X i=0
2i =
n X
2i + 2n+1
Now we can make use of S(n) by substituting its right side, 2n+1 − 1, for in Equation (2.4): n+1 X
(2.4)
i=0
2i = 2n+1 − 1 + 2n+1
Pn
i=0
2i
(2.5)
i=0
When we simplify the right side of Equation (2.5), it becomes 2 × 2n+1 − 1, or 2n+2 − 1. Now we see that the summation on the left side of (2.5) is the same as the left side of (2.3), and the right side of (2.5) is equal to the right side of (2.3). We have thus proved the validity of Equation (2.3) by using the equality S(n); that proof is the inductive step. The conclusion we draw is that S(n) holds for every nonnegative value of n. ✦
Why Does Proof by Induction Work? In an inductive proof, we first prove that S(0) is true. Next we show that if S(n) is true, then S(n + 1) holds. But why can we then conclude that S(n) is true for all n ≥ 0? We shall offer two “proofs.” A mathematician would point out that
SEC. 2.3
INDUCTIVE PROOFS
37
Substituting for Variables People are often confused when they have to substitute for a variable such as n in S(n), an expression involving the same variable. For example, we substituted n + 1 for n in S(n) to get Equation (2.3). To make the substitution, we must first mark every occurrence of n in S. One useful way to do so is to replace n by some new variable — say m — that does not otherwise appear in S. For example, S(n) would become m X 2i = 2m+1 − 1 i=0
We then literally substitute the desired expression, n + 1 in this case, for each occurrence of m. That gives us n+1 X
2i = 2(n+1)+1 − 1
i=0
When we simplify (n + 1) + 1 to n + 2, we have (2.3). Note that we should put parentheses around the expression substituted, to avoid accidentally changing the order of operations. For example, had we substituted n + 1 for m in the expression 2 × m, and not placed the parentheses around n + 1, we would have gotten 2 × n + 1, rather than the correct expression 2 × (n + 1), which equals 2 × n + 2.
each of our “proofs” that induction works requires an inductive proof itself, and therefore is no proof at all. Technically, induction must be accepted as axiomatic. Nevertheless, many people find the following intuition useful. In what follows, we assume that the basis value is n = 0. That is, we know that S(0) is true and that for all n greater than 0, if S(n) is true, then S(n + 1) is true. Similar arguments work if the basis value is any other integer. First “proof”: Iteration of the inductive step. Suppose we want to show that S(a) is true for a particular nonnegative integer a. If a = 0, we just invoke the truth of the basis, S(0). If a > 0, then we argue as follows. We know that S(0) is true, from the basis. The statement “S(n) implies S(n + 1),” with 0 in place of n, says “S(0) implies S(1).” Since we know that S(0) is true, we now know that S(1) is true. Similarly, if we substitute 1 for n, we get “S(1) implies S(2),” and so we also know that S(2) is true. Substituting 2 for n, we have “S(2) implies S(3),” so that S(3) is true, and so on. No matter what the value of a is, we eventually get to S(a), and we are done. Second “proof”: Least counterexample. Suppose S(n) were not true for at least one value of n. Let a be the least nonnegative integer for which S(a) is false. If a = 0, then we contradict the basis, S(0), and so a must be greater than 0. But if a > 0, and a is the least nonnegative integer for which S(a) is false, then S(a − 1) must be true. Now, the inductive step, with n replaced by a−1, tells us that S(a−1) implies S(a). Since S(a−1) is true, S(a) must be true, another contradiction. Since we assumed there were nonnegative values of n for which S(n) is false and derived a contradiction, S(n) must therefore be true for any n ≥ 0.
38
ITERATION, INDUCTION, AND RECURSION
Error-Detecting Codes We shall now begin an extended example of “error-detecting codes,” a concept that is interesting in its own right and also leads to an interesting inductive proof. When we transmit information over a data network, we code characters (letters, digits, punctuation, and so on) into strings of bits, that is, 0’s and 1’s. For the moment let us assume that characters are represented by seven bits. However, it is normal to transmit more than seven bits per character, and an eighth bit can be used to help detect some simple errors in transmission. That is, occasionally, one of the 0’s or 1’s gets changed because of noise during transmission, and is received as the opposite bit; a 0 entering the transmission line emerges as a 1, or vice versa. It is useful if the communication system can tell when one of the eight bits has been changed, so that it can signal for a retransmission. To detect changes in a single bit, we must be sure that no two characters are represented by sequences of bits that differ in only one position. For then, if that position were changed, the result would be the code for the other character, and we could not detect that an error had occurred. For example, if the code for one character is the sequence of bits 01010101, and the code for another is 01000101, then a change in the fourth position from the left turns the former into the latter. One way to be sure that no characters have codes that differ in only one position is to precede the conventional 7-bit code for the character by a parity bit. If the total number of 1’s in a group of bits is odd, the group is said to have odd parity. If the number of 1’s in the group is even, then the group has even parity. The coding scheme we select is to represent each character by an 8-bit code with even parity; we could as well have chosen to use only the codes with odd parity. We force the parity to be even by selecting the parity bit judiciously.
Parity bit
ASCII
✦
Example 2.5. The conventional ASCII (pronounced “ask-ee”; it stands for “American Standard Code for Information Interchange”) 7-bit code for the character A is 1000001. That sequence of seven bits already has an even number of 1’s, and so we prefix it by 0 to get 01000001. The conventional code for C is 1000011, which differs from the 7-bit code for A only in the sixth position. However, this code has odd parity, and so we prefix a 1 to it, yielding the 8-bit code 11000011 with even parity. Note that after prefixing the parity bits to the codes for A and C, we have 01000001 and 11000011, which differ in two positions, namely the first and seventh, as seen in Fig. 2.5. ✦
A: 0 1 0 0 0 0 0 1 C: 1 1 0 0 0 0 1 1 Fig. 2.5. We can choose the initial parity bit so the 8-bit code always has even parity.
We can always pick a parity bit to attach to a 7-bit code so that the number of 1’s in the 8-bit code is even. We pick parity bit 0 if the 7-bit code for the character at hand has even parity, and we pick parity bit 1 if the 7-bit code has odd parity. In either case, the number of 1’s in the 8-bit code is even.
SEC. 2.3
INDUCTIVE PROOFS
39
No two sequences of bits that each have even parity can differ in only one position. For if two such bit sequences differ in exactly one position, then one has exactly one more 1 than the other. Thus, one sequence must have odd parity and the other even parity, contradicting our assumption that both have even parity. We conclude that addition of a parity bit to make the number of 1’s even serves to create an error-detecting code for characters. The parity-bit scheme is quite “efficient,” in the sense that it allows us to transmit many different characters. Note that there are 2n different sequences of n bits, since we may choose either of two values (0 or 1) for the first position, either of two values for the second position, and so on, a total of 2 × 2 × · · · × 2 (n factors) possible strings. Thus, we might expect to be able to represent up to 28 = 256 characters with eight bits. However, with the parity scheme, we can choose only seven of the bits; the eighth is then forced upon us. We can thus represent up to 27 , or 128 characters, and still detect single errors. That is not so bad; we can use 128/256, or half, of the possible 8-bit codes as legal codes for characters, and still detect an error in one bit. Similarly, if we use sequences of n bits, choosing one of them to be the parity bit, we can represent 2n−1 characters by taking sequences of n − 1 bits and prefixing the suitable parity bit, whose value is determined by the other n − 1 bits. Since there are 2n sequences of n bits, we can represent 2n−1 /2n , or half the possible number of characters, and still detect an error in any one of the bits of a sequence. Is it possible to detect errors and use more than half the possible sequences of bits as legal codes? Our next example tells us we cannot. The inductive proof uses a statement that is not true for 0, and for which we must choose a larger basis, namely 1.
✦
Error-detecting code
Example 2.6. We shall prove the following by induction on n. S(n): If C is any set of bit strings of length n that is error detecting (i.e., if there are no two strings that differ in exactly one position), then C contains at most 2n−1 strings.
STATEMENT
This statement is not true for n = 0. S(0) says that any error-detecting set of strings of length 0 has at most 2−1 strings, that is, half a string. Technically, the set C consisting of only the empty string (string with no positions) is an error-detecting set of length 0, since there are no two strings in C that differ in only one position. Set C has more than half a string; it has one string to be exact. Thus, S(0) is false. However, for all n ≥ 1, S(n) is true, as we shall see. The basis is S(1); that is, any error-detecting set of strings of length one has at most 21−1 = 20 = 1 string. There are only two bit strings of length one, the string 0 and the string 1. However, we cannot have both of them in an error-detecting set, because they differ in exactly one position. Thus, every error-detecting set for n = 1 must have at most one string.
BASIS.
Let n ≥ 1, and assume that the inductive hypothesis — an errordetecting set of strings of length n has at most 2n−1 strings — is true. We must
INDUCTION.
40
ITERATION, INDUCTION, AND RECURSION
show, using this assumption, that any error-detecting set C of strings with length n + 1 has at most 2n strings. Thus, divide C into two sets, C0 , the set of strings in C that begin with 0, and C1 , the set of strings in C that begin with 1. For instance, suppose n = 2 and C is the code with strings of length n + 1 = 3 constructed using a parity bit. Then, as shown in Fig. 2.6, C consists of the strings 000, 101, 110, and 011; C0 consists of the strings 000 and 011, and C1 has the other two strings, 101 and 110.
0 00 D 0 11 0
C0 C
1 01 D 1 10 1
C1
Fig. 2.6. The set C is split into C0 , the strings beginning with 0, and C1 , the strings beginning with 1. D0 and D1 are formed by deleting the leading 0’s and 1’s, respectively.
Consider the set D0 consisting of those strings in C0 with the leading 0 removed. In our example above, D0 contains the strings 00 and 11. We claim that D0 cannot have two strings differing in only one bit. The reason is that if there are two such strings — say a1 a2 · · · an and b1 b2 · · · bn — then restoring their leading 0’s gives us two strings in C0 , 0a1 a2 · · · an and 0b1 b2 · · · bn , and these strings would differ in only one position as well. But strings in C0 are also in C, and we know that C does not have two strings that differ in only one position. Thus, neither does D0 , and so D0 is an error detecting set. Now we can apply the inductive hypothesis to conclude that D0 , being an error-detecting set with strings of length n, has at most 2n−1 strings. Thus, C0 has at most 2n−1 strings. We can reason similarly about the set C1 . Let D1 be the set of strings in C1 , with their leading 1’s deleted. D1 is an error-detecting set with strings of length n, and by the inductive hypothesis, D1 has at most 2n−1 strings. Thus, C1 has at most 2n−1 strings. However, every string in C is in either C0 or C1 . Therefore, C has at most 2n−1 + 2n−1 , or 2n strings. We have proved that S(n) implies S(n + 1), and so we may conclude that S(n) is true for all n ≥ 1. We exclude n = 0 from the claim, because the basis is n = 1, not n = 0. We now see that the error-detecting sets constructed by parity check are as large as possible, since they have exactly 2n−1 strings when strings of n bits are used. ✦
SEC. 2.3
INDUCTIVE PROOFS
41
How to Invent Inductive Proofs There is no “crank to turn” that is guaranteed to give you an inductive proof of any (true) statement S(n). Finding inductive proofs, like finding proofs of any kind, or like writing programs that work, is a task with intellectual challenge, and we can only offer a few words of advice. If you examine the inductive steps in Examples 2.4 and 2.6, you will notice that in each case we had to rework the statement S(n + 1) that we were trying to prove so that it incorporated the inductive hypothesis, S(n), plus something extra. In Example 2.4, we expressed the sum 1 + 2 + 4 + · · · + 2n + 2n+1 as the sum 1 + 2 + 4 + · · · + 2n which the inductive hypothesis tells us something about, plus the extra term, 2n+1 . In Example 2.6, we expressed the set C, with strings of length n + 1, in terms of two sets of strings (which we called D0 and D1 ) of length n, so that we could apply the inductive hypothesis to these sets and conclude that both of these sets were of limited size. Of course, working with the statement S(n + 1) so that we can apply the inductive hypothesis is just a special case of the more universal problem-solving adage “Use what is given.” The hard part always comes when we must deal with the “extra” part of S(n+1) and complete the proof of S(n+1) from S(n). However, the following is a universal rule: ✦
An inductive proof must at some point say “· · ·and by the inductive hypothesis we know that· · · .” If it doesn’t, then it isn’t a inductive proof.
EXERCISES 2.3.1: Show the following formulas by induction on n starting at n = 1. Pn a) i=1 i = n(n + 1)/2. Pn 2 b) i=1 i = n(n + 1)(2n + 1)/6. Pn 3 2 2 c) i=1 i = n (n + 1) /4. Pn d) i=1 1/i(i + 1) = n/(n + 1). Triangular number
2.3.2: Numbers of the form tn = n(n + 1)/2 are called triangular numbers, because Pn marbles arranged in an equilateral triangle, n on a side, will total i=1 i marbles, which we saw in Exercise 2.3.1(a) is tn marbles. For example, bowling pins are arranged in a triangle P4 on a side and there are t4 = 4 × 5/2 = 10 pins. Show by induction on n that nj=1 tj = n(n + 1)(n + 2)/6. 2.3.3: Identify the parity of each of the following bit sequences as even or odd: a) b)
01101 111000111
42
ITERATION, INDUCTION, AND RECURSION
c)
010101
2.3.4: Suppose we use three digits — say 0, 1, and 2 — to code symbols. A set of strings C formed from 0’s, 1’s, and 2’s is error detecting if no two strings in C differ in only one position. For example, {00, 11, 22} is an error-detecting set with strings of length two, using the digits 0, 1, and 2. Show that for any n ≥ 1, an error-detecting set of strings of length n using the digits 0, 1, and 2, cannot have more than 3n−1 strings. 2.3.5*: Show that for any n ≥ 1, there is an error-detecting set of strings of length n, using the digits 0, 1, and 2, that has 3n−1 strings. 2.3.6*: Show that if we use k symbols, for any k ≥ 2, then there is an errordetecting set of strings of length n, using k different symbols as “digits,” with k n−1 strings, but no such set of strings with more than k n−1 strings. 2.3.7*: If n ≥ 1, the number of strings using the digits 0, 1, and 2, with no two consecutive places holding the same digit, is 3 × 2n−1 . For example, there are 12 such strings of length three: 010, 012, 020, 021, 101, 102, 120, 121, 201, 202, 210, and 212. Prove this claim by induction on the length of the strings. Is the formula true for n = 0? 2.3.8*: Prove that the ripple-carry addition algorithm discussed in Section 1.3 produces the correct answer. Hint : Show by induction on i that after considering the first i places from the right end, the sum of the tails of length i for the two addends equals the number whose binary representation is the carry bit followed by the i bits of answer generated so far. 2.3.9*: The formula for the sum of n terms of a geometric series a, ar, ar2 , . . . , arn−1 is n−1 X i=0
ari =
(arn − a) (r − 1)
Prove this formula by induction on n. Note that you must assume r 6= 1 for the formula to hold. Where do you use that assumption in your proof? 2.3.10: The formula for the sum of an arithmetic series with first term a and increment b, that is, a, (a + b), (a + 2b), . . . , a + (n − 1)b , is n−1 X i=0
a + bi = n 2a + (n − 1)b /2
a)
Prove this formula by induction on n.
b)
Show how Exercise 2.3.1(a) is an example of this formula.
2.3.11: Give two informal proofs that induction starting at 1 “works,” although the statement S(0) may be false. 2.3.12: Show by induction on the length of strings that the code consisting of the odd-parity strings detects errors.
SEC. 2.3
INDUCTIVE PROOFS
43
Arithmetic and Geometric Sums There are two formulas from high-school algebra that we shall use frequently. They each have interesting inductive proofs, which we ask the reader to provide in Exercises 2.3.9 and 2.3.10. Arithmetic series
An arithmetic series is a sequence of n numbers of the form a, (a + b), (a + 2b), . . . , a + (n − 1)b The first term is a, and each term is b larger than the one before. The sum of these n numbers is n times the average of the first and last terms; that is: n−1 X i=0
a + bi = n 2a + (n − 1)b /2
For example, consider the sum of 3 + 5 + 7 + 9 + 11. There are n = 5 terms, the first is 3 and the last 11. Thus, the sum is 5 × (3 + 11)/2 = 5 × 7 = 35. You can check that this sum is correct by adding the five integers. Geometric series
A geometric series is a sequence of n numbers of the form a, ar, ar2 , ar3 , . . . , arn−1 That is, the first term is a, and each successive term is r times the previous term. The formula for the sum of n terms of a geometric series is n−1 X i=0
ari =
(arn − a) (r − 1)
Here, r can be greater or less than 1. If r = 1, the above formula does not work, but all terms are a so the sum is obviously an. As an example of a geometric series sum, consider 1 + 2 + 4 + 8 + 16. Here, n = 5, the first term a is 1, and the ratio r is 2. Thus, the sum is (1 × 25 − 1)/(2 − 1) = (32 − 1)/1 = 31 as you may check. For another example, consider 1 + 1/2 + 1/4 + 1/8 + 1/16. Again n = 5 and a = 1, but r = 1/2. The sum is 15 1 × ( 21 )5 − 1 /( 21 − 1) = (−31/32)/(−1/2) = 1 16
Error-correcting code
2.3.13**: If no two strings in a code differ in fewer than three positions, then we can actually correct a single error, by finding the unique string in the code that differs from the received string in only one position. It turns out that there is a code of 7-bit strings that corrects single errors and contains 16 strings. Find such a code. Hint : Reasoning it out is probably best, but if you get stuck, write a program that searches for such a code. 2.3.14*: Does the even parity code detect any “double errors,” that is, changes in two different bits? Can it correct any single errors?
44
ITERATION, INDUCTION, AND RECURSION
Template for Simple Inductions Let us summarize Section 2.3 by giving a template into which the simple inductions of that section fit. Section 2.4 will cover a more general template.
✦ ✦ ✦ ✦
2.4
Strong and weak induction
1.
Specify the statement S(n) to be proved. Say you are going to prove S(n) by induction on n, for all n ≥ i0 . Here, i0 is the constant of the basis; usually i0 is 0 or 1, but it could be any integer. Explain intuitively what n means, e.g., the length of codewords.
2.
State the basis case, S(i0 ).
3.
Prove the basis case. That is, explain why S(i0 ) is true.
4.
Set up the inductive step by stating that you are assuming S(n) for some n ≥ i0 , the “inductive hypothesis.” Express S(n + 1) by substituting n + 1 for n in the statement S(n).
5.
Prove S(n + 1), assuming the inductive hypothesis S(n).
6.
Conclude that S(n) is true for all n ≥ i0 (but not necessarily for smaller n).
Complete Induction In the examples seen so far, we have proved that S(n + 1) is true using only S(n) as an inductive hypothesis. However, since we prove our statement S for values of its parameter starting at the basis value and proceeding upward, we are entitled to use S(i) for all values of i, from the basis value up to n. This form of induction is called complete (or sometimes perfect or strong) induction, while the simple form of induction of Section 2.3, where we used only S(n) to prove S(n + 1) is sometimes called weak induction. Let us begin by considering how to perform a complete induction starting with basis n = 0. We prove that S(n) is true for all n ≥ 0 in two steps: 1.
We first prove the basis, S(0).
2.
As an inductive hypothesis, we assume all of S(0), S(1), . . . , S(n) to be true. From these statements we prove that S(n + 1) holds.
As for weak induction described in the previous section, we can also pick some value a other than 0 as the basis. Then, for the basis we prove S(a), and in the inductive step we are entitled to assume only S(a), S(a + 1), . . . , S(n). Note that weak induction is a special case of complete induction in which we elect not to use any of the previous statements except S(n) to prove S(n + 1). Figure 2.7 suggests how complete induction works. Each instance of the statement S(n) can (optionally) use any of the lower-indexed instances to its right in its proof.
SEC. 2.4
S(n)
···
S(2)
COMPLETE INDUCTION
S(1)
45
S(0)
Fig. 2.7. Complete induction allows each instance to use one, some, or all of the previous instances in its proof.
Inductions With More Than One Basis Case When performing a complete induction, there are times when it is useful to have more than one basis case. If we wish to prove a statement S(n) for all n ≥ i0 , then we could treat not only i0 as a basis case, but also some number of consecutive integers above i0 , say i0 , i0 + 1, i0 + 2, . . . , j0 . Then we must do the following two steps:
✦
1.
Prove each of the basis cases, the statements S(i0 ), S(i0 + 1), . . . , S(j0 ).
2.
As an inductive hypothesis, assume all of S(i0 ), S(i0 + 1), . . . , S(n) hold, for some n ≥ j0 , and prove S(n + 1).
Example 2.7. Our first example of a complete induction is a simple one that uses multiple basis cases. As we shall see, it is only “complete” in a limited sense. To prove S(n + 1) we do not use S(n) but we use S(n − 1) only. In more general complete inductions to follow, we use S(n), S(n − 1), and many other instances of the statement S. Let us prove by induction on n the following statement for all n ≥ 0.4
S(n): There are integers a and b (positive, negative, or 0) such that n = 2a + 3b.
STATEMENT
BASIS.
i)
We shall take both 0 and 1 as basis cases.
For n = 0 we may pick a = 0 and b = 0. Surely 0 = 2 × 0 + 3 × 0.
ii) For n = 1, pick a = −1 and b = 1. Then 1 = 2 × (−1) + 3 × 1. INDUCTION. Now, we may assume S(n) and prove S(n + 1), for any n ≥ 1. Note that we may assume n is at least the largest of the consecutive values for which we have proved the basis: n ≥ 1 here. Statement S(n + 1) says that n + 1 = 2a + 3b for some integers a and b. The inductive hypothesis says that all of S(0), S(1), . . . , S(n) are true. Note that we begin the sequence at 0 because that was the lowest of the consecutive basis cases. Since n ≥ 1 can be assumed, we know that n − 1 ≥ 0, and therefore, S(n − 1) is true. This statement says that there are integers a and b such that n−1 = 2a+3b. 4
Actually, this statement is true for all n, positive or negative, but the case of negative n requires a second induction which we leave as an exercise.
46
ITERATION, INDUCTION, AND RECURSION
Since we need a in the statement S(n+1), let us restate S(n−1) to use different names for the integers and say there are integers a′ and b′ such that n − 1 = 2a′ + 3b′
(2.6)
If we add 2 to both sides of (2.6), we have n + 1 = 2(a′ + 1) + 3b′ . If we then let a = a′ + 1 and b = b′ , we have the statement n + 1 = 2a + 3b for some integers a and b. This statement is S(n + 1), so we have proved the induction. Notice that in this proof, we did not use S(n), but we did use S(n − 1). ✦
Justifying Complete Induction Like the ordinary or “weak” induction discussed in Section 2.3, complete induction can be justified intuitively as a proof technique by a “least counterexample” argument. Let the basis cases be S(i0 ), S(i0 + 1), . . . , S(j0 ), and suppose we have shown that for any n ≥ j0 , S(i0 ), S(i0 +1), . . . , S(n) together imply S(n+1). Now, suppose S(n) were not true for at least one value of n ≥ i0 , and let b be the smallest integer equal to or greater than i0 for which S(b) is false. Then b cannot be between i0 and j0 , or the basis is contradicted. Further, b cannot be greater than j0 . If it were, all of S(i0 ), S(i0 + 1), . . . , S(b − 1) would be true. But the inductive step would then tell us that S(b) is true, yielding the contradiction.
Normal Forms for Arithmetic Expressions We shall now explore an extended example concerning the transformation of arithmetic expressions to equivalent forms. It offers an illustration of a complete induction that takes full advantage of the fact that the statement S to be proved may be assumed for all arguments from n downward. By way of motivation, a compiler for a programming language may take advantage of the algebraic properties of arithmetic operators to rearrange the order in which the operands of an arithmetic expression are evaluated. The goal is of this rearrangement is to find a way for the computer to evaluate the expression using less time than the obvious evaluation order takes. In this section we consider arithmetic expressions containing a single associative and commutative operator, like +, and examine what rearrangements of operands are possible. We shall prove that if we have any expression involving only the operator +, then the value of the expression is equal to the value of any other expression with + applied to the same operands, ordered and/or grouped in any arbitrary way. For example, a3 + (a4 + a1 ) + (a2 + a5 ) = a1 + a2 + a3 + (a4 + a5 ) We shall prove this claim by performing two separate inductions, the first of which is a complete induction.
✦
Example 2.8. We shall prove by complete induction on n (the number of operands in an expression) the statement
SEC. 2.4
COMPLETE INDUCTION
47
Associativity and Commutativity Associative law
Recall that the associative law for addition says that we can add three values either by adding the first two and then adding the third to the result, or by adding the first to the result of adding the second and third; the result will be the same. Formally, (E1 + E2 ) + E3 = E1 + (E2 + E3 ) where E1 , E2 , and E3 are any arithmetic expressions. For instance, (1 + 2) + 3 = 1 + (2 + 3) Here, E1 = 1, E2 = 2, and E3 = 3. Also, (xy) + (3z − 2) + (y + z) = xy + (3z − 2) + (y + z)
Commutative law
Here, E1 = xy, E2 = 3z − 2, and E3 = y + z. Also recall that the commutative law for addition says that we can sum two expressions in either order. Formally, E1 + E2 = E2 + E1 For example, 1 + 2 = 2 + 1, and xy + (3z − 2) = (3z − 2) + xy.
S(n): If E is an expression involving the operator + and n operands, and a is one of those operands, then E can be transformed, by using the associative and commutative laws, into an expression of the form a + F , where F is an expression involving all the operands of E except a, grouped in some order using the operator +.
STATEMENT
Statement S(n) only holds for n ≥ 2, since there must be at least one occurrence of the operator + in E. Thus, we shall use n = 2 as our basis. BASIS. Let n = 2. Then E can be only a + b or b + a, for some operand b other than a. In the first case, we let F be the expression b, and we are done. In the second case, we note that by the commutative law, b + a can be transformed into a + b, and so we may again let F = b.
Let E have n + 1 operands, and assume that S(i) is true for i = 2, 3, . . . , n. We need to prove the inductive step for n ≥ 2, so we may assume that E has at least three operands and therefore at least two occurrences of +. We can write E as E1 + E2 for some expressions E1 and E2 . Since E has exactly n + 1 operands, and E1 and E2 must each have at least one of these operands, it follows that neither E1 nor E2 can have more than n operands. Thus, the inductive hypothesis applies to E1 and E2 , as long as they have more than one operand each (because we started with n = 2 as the basis). There are four cases we must consider, depending whether a is in E1 or E2 , and on whether it is or is not the only operand in E1 or E2 . INDUCTION.
a)
E1 is a by itself. An example of this case occurs when E is a + (b + c); here E1 is a and E2 is b + c. In this case, E2 serves as F ; that is, E is already of the form a + F .
48
ITERATION, INDUCTION, AND RECURSION
b)
E1 has more than one operand, and a is among them. For instance, E = c + (d + a) + (b + e) where E1 = c + (d + a) and E2 = b + e. Here, since E1 has no more than n operands but at least two operands, we can apply the inductive hypothesis to tell us that E1 can be transformed, using the commutative and associative laws, into a+E3 . Thus, E can be transformed into (a+E3 )+E2 . We apply the associative law and see that E can further be transformed into a + (E3 + E2 ). Thus, we may choose F to be E3 + E2 , which proves the inductive step in this case. For our example E above, we may suppose that E2 = c + (d + a) is transformed by the inductive hypothesis into a + (c + d). Then E can be regrouped into a + (c + d) + (b + e) .
c)
E2 is a alone. For instance, E = (b+c)+a. In this case, we use the commutative law to transform E into a + E1 , which is of the desired form if we let F be E1 .
d)
E2 has more than one operand, including a. An example is E = b + (a + c). Apply the commutative law to transform E into E2 + E1 . Then proceed as in case (b). If E = b + (a + c), we transform E first into (a + c) + b. By the inductive hypothesis, a + c can be put in the desired form; in fact, it is already there. The associative law then transforms E into a + (c + b).
In all four cases, we have transformed E to the desired form. Thus, the inductive step is proved, and we conclude that S(n) for all n ≥ 2. ✦
✦
Example 2.9. The inductive proof of Example 2.8 leads directly to an algorithm that puts an expression into the desired form. As an example, consider the expression E = x + (z + v) + (w + y) and suppose that v is the operand we wish to “pull out,” that is, to play the role of a in the transformation of Example 2.8. Initially, we have an example of case (b), with E1 = x + (z + v), and E2 = w + y. Next, we must work on the expression E1 and “pull out” v. E1 is an example of case (d), and so we first apply the commutative law to transform it into (z + v) + x. As an instance of case (b), we must work on the expression z + v, which is an instance of case (c). We thus transform it by the commutative law into v + z. Now E1 has been transformed into (v+z)+x, and a further use of the associative law transforms it to v+(z+x). That, in turn, transforms E into v+(z+x) +(w+y). By the associative law, E can be transformed into v + (z + x) + (w + y) . Thus, E = v + F , where F is the expression (z + x) + (w + y). The entire sequence of transformations is summarized in Fig. 2.8. ✦ Now, we can use the statement proved in Example 2.8 to prove our original contention, that any two expressions involving the operator + and the same list of distinct operands can be transformed one to the other by the associative and commutative laws. This proof is by weak induction, as discussed in Section 2.3, rather than complete induction.
SEC. 2.4
COMPLETE INDUCTION
49
x + (z + v) + (w + y) (z + v) + x + (w + y) (v + z) + x + (w + y) v + (z + x) + (w + y) v + (z + x) + (w + y) Fig. 2.8. Using the commutative and associative laws, we can “pull out” any operand, such as v.
✦
Example 2.10. Let us prove the following statement by induction on n, the number of operands in an expression.
T (n): If E and F are expressions involving the operator + and the same set of n distinct operands, then it is possible to transform E into F by a sequence of applications of the associative and commutative laws.
STATEMENT
If n = 1, then the two expressions must both be a single operand a. Since they are the same expression, surely E is “transformable” into F .
BASIS.
Suppose T (n) is true, for some n ≥ 1. We shall now prove T (n + 1). Let E and F be expressions involving the same set of n+1 operands, and let a be one of these operands. Since n + 1 ≥ 2, S(n + 1) — the statement from Example 2.8 — must hold. Thus, we can transform E into a + E1 for some expression E1 involving the other n operands of E. Similarly, we can transform F into a + F1 , for some expression F1 involving the same n operands as E1 . What is more important, in this case, is that we can also perform the transformations in the opposite direction, transforming a + F1 into F by use of the associative and commutative laws. Now we invoke the inductive hypothesis T (n) on the expressions E1 and F1 . Each has the same n operands, and so the inductive hypothesis applies. That tells us we can transform E1 into F1 , and therefore we can transform a + E1 into a + F1 . We may thus perform the transformations INDUCTION.
E → · · · → a + E1 Using S(n + 1) → · · · → a + F1 Using T (n) → ··· → F Using S(n + 1) in reverse to turn E into F . ✦
✦
Example 2.11. Let us transform E = (x+y)+(w+z) into F = (w+z)+y +x.
We begin by selecting an operand, say w, to “pull out.” If we check the cases in Example 2.8, we see that for E we perform the sequence of transformations (x + y) + (w + z) → (w + z) + (x + y) → w + z + (x + y) (2.7) while for F we do (w + z) + y + x → w + (z + y) + x → w + (z + y) + x
(2.8)
50
ITERATION, INDUCTION, AND RECURSION
We now have the subproblem of transforming z + (x + y) into (z + y) + x. We shall do so by “pulling out” x. The sequences of transformations are z + (x + y) → (x + y) + z → x + (y + z)
(2.9)
and (z + y) + x → x + (z + y)
(2.10)
That, in turn, gives us a subproblem of transforming y + z into z + y. We do so by an application of the commutative law. Strictly speaking, we use the technique of Example 2.8 to “pull out” y for each, leaving y + z for each expression. Then the basis case for Example 2.10 tells us that the expression z can be “transformed” into itself. We can now transform z + (x + y) into (z + y) + x by the steps of line (2.9), then applying the commutative law to subexpression y + z, and finally using the transformation of line (2.10), in reverse. We use these transformations as the middle part of the transformation from (x + y) + (w + z) to (w + z) + y + x. First we apply the transformations of line (2.7), and then the transformations just discussed to change z + (x + y) into (z + y) + x, and finally the transformations of line (2.8) in reverse. The entire sequence of transformations is summarized in Fig. 2.9. ✦ (x + y) + (w + z) (w + z) + (x + y) w + z + (x + y) w + (x + y) + z w + x + (y + z) w + x + (z + y) w + (z + y)+ x w + (z + y) + x (w + z) + y + x
Expression E Middle of (2.7) End of (2.7) Middle of (2.9) End of (2.9) Commutative law (2.10) in reverse Middle of (2.8) in reverse Expression F , end of (2.8) in reverse
Fig. 2.9. Transforming one expression into another using the commutative and associative laws.
EXERCISES 2.4.1: “Pull out” from the expression E = (u + v) + w + (x + y) + z each of the operands in turn. That is, start from E in each of the six parts, and use the techniques of Example 2.8 to transform E into an expression of the form u + E1 . Then transform E1 into an expression of the form v + E2 , and so on. 2.4.2: Use the technique of Example 2.10 to transform a) w + x + (y + z) into (w + x) + y + z b) (v + w) + (x + y) + z into (y + w) + (v + z) + x Binary operator
2.4.3*: Let E be an expression with operators +, −, ∗, and /; each operator is binary only; that is, it takes two operands. Show, using a complete induction on the number of occurrences of operators in E, that if E has n operator occurrences, then E has n + 1 operands.
SEC. 2.4
COMPLETE INDUCTION
51
A Template for All Inductions The following organization of inductive proofs covers complete inductions with multiple basis cases. As a special case it includes the weak inductions of Section 2.3, and it includes the common situation where there is only one basis case. 1.
Specify the statement S(n) to be proved. Say that you are going to prove S(n) by induction on n, for n ≥ i0 . Specify what i0 is; often it is 0 or 1, but i0 could be any integer. Explain intuitively what n represents.
2.
State the basis case(s). These will be all the integers from i0 up to some integer j0 . Often j0 = i0 , but j0 could be larger.
3.
Prove each of the basis cases S(i0 ), S(i0 + 1), . . . , S(j0 ).
4.
Set up the inductive step by stating that you are assuming S(i0 ), S(i0 + 1), . . . , S(n) (the “inductive hypothesis”) and that you want to prove S(n + 1). State that you are assuming n ≥ j0 ; that is, n is at least as great as the highest basis case. Express S(n + 1) by substituting n + 1 for n in the statement S(n).
5.
Prove S(n + 1) under the assumptions mentioned in (4). If the induction is a weak, rather than complete, induction, then only S(n) will be used in the proof, but you are free to use any or all of the statements of the inductive hypothesis.
6.
Conclude that S(n) is true for all n ≥ i0 (but not necessarily for smaller n).
2.4.4: Give an example of a binary operator that is commutative but not associative. 2.4.5: Give an example of a binary operator that is associative but not commutative. 2.4.6*: Consider an expression E whose operators are all binary. The length of E is the number of symbols in E, counting an operator or a left or right parenthesis as one symbol, and also counting any operand such as 123 or abc as one symbol. Prove that E must have an odd length. Hint: Prove the claim by complete induction on the length of the expression E. 2.4.7: Show that every negative integer can be written in the form 2a + 3b for some (not necessarily positive) integers a and b. 2.4.8*: Show that every integer (positive or negative) can be written in the form 5a + 7b for some (not necessarily positive) integers a and b. 2.4.9*: Is every proof by weak induction (as in Section 2.3) also a proof by complete induction? Is every proof by complete induction also a proof by weak induction? 2.4.10*: We showed in this section how to justify complete induction by a least counterexample argument. Show how complete induction can also be justified by an iteration.
52
ITERATION, INDUCTION, AND RECURSION
Truth in Advertising There are many difficulties, both theoretical and practical, in proving programs correct. An obvious question is “What does it mean for a program to be ‘correct’ ?” As we mentioned in Chapter 1, most programs in practice are written to satisfy some informal specification. The specification itself may be incomplete or inconsistent. Even if there were a precise formal specification, we can show that no algorithm exists to prove that an arbitrary program is equivalent to a given specification. However, in spite of these difficulties, it is beneficial to state and prove assertions about programs. The loop invariants of a program are often the most useful short explanation one can give of how the program works. Further, the programmer should have a loop invariant in mind while writing a piece of code. That is, there must be a reason why a program works, and this reason often has to do with an inductive hypothesis that holds each time the program goes around a loop or each time it performs a recursive call. The programmer should be able to envision a proof, even though it may be impractical to write out such a proof line by line.
✦ ✦ ✦ ✦
2.5
Proving Properties of Programs In this section we shall delve into an area where inductive proofs are essential: proving that a program does what it is claimed to do. We shall see a technique for explaining what an iterative program does as it goes around a loop. If we understand what the loops do, we generally understand what we need to know about an iterative program. In Section 2.9, we shall consider what is needed to prove properties of recursive programs.
Loop Invariants The key to proving a property of a loop in a program is selecting a loop invariant, or inductive assertion, which is a statement S that is true each time we enter a particular point in the loop. The statement S is then proved by induction on a parameter that in some way measures the number of times we have gone around the loop. For example, the parameter could be the number of times we have reached the test of a while-loop, it could be the value of the loop index in a for-loop, or it could be some expression involving the program variables that is known to increase by 1 for each time around the loop.
Inductive assertion
✦
Example 2.12. As an example, let us consider the inner loop of SelectionSort from Section 2.2. These lines, with the original numbering from Fig. 2.2, are (2) (3) (4) (5)
small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j;
SEC. 2.5
PROVING PROPERTIES OF PROGRAMS
53
small = i
j = i+1
S(k)
j < n?
no
yes loop body lines (4) and (5)
j = j+1
Fig. 2.10. Flowchart for the inner loop of SelectionSort.
Recall that the purpose of these lines is to make small equal to the index of an element of A[i..n-1] with the smallest value. To see why that claim is true, consider the flowchart for our loop shown in Fig. 2.10. This flowchart shows the five steps necessary to execute the program: 1.
First, we need to initialize small to i, as we do in line (2).
2.
At the beginning of the for-loop of line (3), we need to initialize j to i + 1.
3.
Then, we need to test whether j < n.
4.
If so, we execute the body of the loop, which consists of lines (4) and (5).
5.
At the end of the body, we need to increment j and go back to the test.
In Fig. 2.10 we see a point just before the test that is labeled by a loop-invariant statement we have called S(k); we shall discover momentarily what this statement must be. The first time we reach the test, j has the value i + 1 and small has the value i. The second time we reach the test, j has the value i + 2, because j has been incremented once. Because the body (lines 4 and 5) sets small to i + 1 if A[i + 1] is less than A[i], we see that small is the index of whichever of A[i] and A[i + 1] is smaller.5 5
In case of a tie, small will be i. In general, we shall pretend that no ties occur and talk about “the smallest element” when we really mean “the first occurrence of the smallest element.”
54
ITERATION, INDUCTION, AND RECURSION
Similarly, the third time we reach the test, the value of j is i + 3 and small is the index of the smallest of A[i..i+2]. We shall thus try to prove the following statement, which appears to be the general rule.
S(k): If we reach the test for j < n in the for-statement of line (3) with k as the value of loop index j, then the value of small is the index of the smallest of A[i..k-1].
STATEMENT
Note that we are using the letter k to stand for one of the values that the variable j assumes, as we go around the loop. That is less cumbersome than trying to use j as the value of j, because we sometimes need to keep k fixed while the value of j changes. Also notice that S(k) has the form “if we reach · · · ,” because for some values of k we may never reach the loop test, as we broke out of the loop for a smaller value of the loop index j. If k is one of those values, then S(k) is surely true, because any statement of the form “if A then B” is true when A is false. The basis case is k = i + 1, where i is the value of the variable i at line (3).6 Now j = i + 1 when we begin the loop. That is, we have just executed line (2), which gives small the value i, and we have initialized j to i + 1 to begin the loop. S(i + 1) says that small is the index of the smallest element in A[i..i], which means that the value of small must be i. But we just observed that line (2) causes small to have the value i. Technically, we must also show that j can never have value i + 1 except the first time we reach the test. The reason, intuitively, is that each time around the loop, we increment j, so it will never again be as low as i + 1. (To be perfectly precise, we should give an inductive proof of the assumption that j > i + 1 except the first time through the test.) Thus, the basis, S(i + 1), has been shown to be true.
BASIS.
Now let us assume as our inductive hypothesis that S(k) holds, for some k ≥ i + 1, and prove S(k + 1). First, if k ≥ n, then we break out of the loop when j has the value k, or earlier, and so we are sure never to reach the loop test with the value of j equal to k + 1. In that case, S(k + 1) is surely true. Thus, let us assume that k < n, so that we actually make the test with j equal to k + 1. S(k) says that small indexes the smallest of A[i..k-1], and S(k + 1) says that small indexes the smallest of A[i..k]. Consider what happens in the body of the loop (lines 4 and 5) when j has the value k; there are two cases, depending on whether the test of line (4) is true or not.
INDUCTION.
1.
If A[k] is not smaller than the smallest of A[i..k-1], then the value of small does not change. In that case, however, small also indexes the smallest of A[i..k], since A[k] is not the smallest. Thus, the conclusion of S(k + 1) is true in this case.
2.
If A[k] is smaller than the smallest of A[i] through A[k − 1], then small is set to k. Again, the conclusion of S(k + 1) now holds, because k is the index of the smallest of A[i..k]. 6
As far as the loop of lines (3) to (5) is concerned, i does not change. Thus, i + 1 is an appropriate constant to use as the basis value.
SEC. 2.5
PROVING PROPERTIES OF PROGRAMS
55
Thus, in either case, small is the index of the smallest of A[i..k]. We go around the for-loop by incrementing the variable j. Thus, just before the loop test, when j has the value k + 1, the conclusion of S(k + 1) holds. We have now shown that S(k) implies S(k + 1). We have completed the induction and conclude that S(k) holds for all values k ≥ i + 1. Next, we apply S(k) to make our claim about the inner loop of lines (3) through (5). We exit the loop when the value of j reaches n. Since S(n) says that small indexes the smallest of A[i..n-1], we have an important conclusion about the working of the inner loop. We shall see how it is used in the next example. ✦
(1) (2) (3) (4) (5) (6) (7) (8)
for (i = 0; i < n-1; i++) { small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j; temp = A[small]; A[small] = A[i]; A[i] = temp; } Fig. 2.11. The body of the SelectionSort function.
✦
Example 2.13. Now, let us consider the entire SelectionSort function, the heart of which we reproduce in Fig. 2.11. A flowchart for this code is shown in Fig. 2.12, where “body” refers to lines (2) through (8) of Fig. 2.11. Our inductive assertion, which we refer to as T (m), is again a statement about what must be true just before the test for termination of the loop. Informally, when i has the value m, we have selected m of the smallest elements and sorted them at the beginning of the array. More precisely, we prove the following statement T (m) by induction on m. T (m): If we reach the loop test i < n − 1 of line (1) with the value of variable i equal to m, then
STATEMENT
a)
A[0..m-1] are in sorted order; that is, A[0] ≤ A[1] ≤ · · · ≤ A[m − 1].
b)
All of A[m..n-1] are at least as great as any of A[0..m-1].
The basis case is m = 0. The basis is true for trivial reasons. If we look at the statement T (0), part (a) says that A[0..-1] are sorted. But there are no elements in the range A[0], . . . , A[−1], and so (a) must be true. Similarly, part (b) of T (0) says that all of A[0..n-1] are at least as large as any of A[0..-1]. Since there are no elements of the latter description, part (b) is also true.
BASIS.
56
ITERATION, INDUCTION, AND RECURSION
i = 0
T (m)
i < n − 1?
no
yes loop body lines (2) — (8)
i = i+1
Fig. 2.12. Flow-chart for the entire selection sort function. INDUCTION. For the inductive step, we assume that T (m) is true for some m ≥ 0, and we show that T (m + 1) holds. As in Example 2.12, we are trying to prove a statement of the form “if A then B,” and such a statement is true whenever A is false. Thus, T (m + 1) is true if the assumption that we reach the for-loop test with i equal to m + 1 is false. Thus, we may assume that we actually reach the test with i having the value m + 1; that is, we may assume m < n − 1. When i has the value m, the body of the loop finds a smallest element in A[m..n-1] (as proved by the statement S(m) of Example 2.12). This element is swapped with A[m] in lines (6) through (8). Part (b) of the inductive hypothesis, T (m), tells us the element chosen must be at least as large as any of A[0..m-1]. Moreover, those elements were sorted, so now all of A[i..m] are sorted. That proves part (a) of statement T (m + 1). To prove part (b) of T (m+ 1), we see that A[m] was just selected to be as small as any of A[m+1..n-1]. Part (a) of T (m) tells us that A[0..m-1] were already as small as any of A[m+1..n-1]. Thus, after executing the body of lines (2) through (8) and incrementing i, we know that all of A[m+1..n-1] are at least as large as any of A[0..m]. Since now the value of i is m + 1, we have shown the truth of the statement T (m + 1) and thus have proved the inductive step.
Now, let m = n − 1. We know that we exit the outer for-loop when i has the value n − 1, so T (n − 1) will hold after we finish this loop. Part (a) of T (n − 1) says that all of A[0..n-2] are sorted, and part (b) says that A[n − 1] is as large as any of the other elements. Thus, after the program terminates the elements in A are in nonincreasing order; that is, they are sorted. ✦
SEC. 2.5
PROVING PROPERTIES OF PROGRAMS
57
Loop Invariants for While-Loops When we have a while-loop of the form while () it usually makes sense to find the appropriate loop invariant for the point just before the test of the condition. Generally, we try to prove the loop invariant holds by induction on the number of times around the loop. Then, when the condition becomes false, we can use the loop invariant, together with the falsehood of the condition, to conclude something useful about what is true after the while-loop terminates. However, unlike for-loops, there may not be a variable whose value counts the number of times around the while-loop. Worse, while the for-loop is guaranteed to iterate only up to the limit of the loop (for example, up to n − 1 for the inner loop of the SelectionSort program), there is no reason to believe that the condition of the while-loop will ever become false. Thus, part of the proof of correctness for a while-loop is a proof that it eventually terminates. We usually prove termination by identifying some expression E, involving the variables of the program, such that
While-loop termination
✦ Factorial
1.
The value of E decreases by at least 1 each time around the loop, and
2.
The loop condition is false if E is as low as some specified constant, such as 0.
Example 2.14. The factorial function, written n!, is defined as the product of the integers 1 × 2 × · · · × n. For example, 1! = 1, 2! = 1 × 2 = 2, and 5! = 1 × 2 × 3 × 4 × 5 = 120 Figure 2.13 shows a simple program fragment to compute n! for integers n ≥ 1. (1) (2) (3) (4) (5) (6) (7)
scanf("%d", &n); i = 2; fact = 1; while (i <= n) { fact = fact*i; i++; } printf("%d\n", fact); Fig. 2.13. Factorial program fragment.
To begin, let us prove that the while-loop of lines (4) to (6) in Fig. 2.13 must terminate. We shall choose E to be the expression n − i. Notice that each time around the while-loop, i is increased by 1 at line (6) and n remains unchanged. Therefore, E decreases by 1 each time around the loop. Moreover, when E is −1 or less, we have n − i ≤ −1, or i ≥ n + 1. Thus, when E becomes negative, the loop condition i ≤ n will be false and the loop will terminate. We don’t know how large E is initially, since we don’t know what value of n will be read. Whatever that value is, however, E will eventually reach as low as −1, and the loop will terminate.
58
ITERATION, INDUCTION, AND RECURSION
Now we must prove that the program of Fig. 2.13 does what it is intended to do. The appropriate loop-invariant statement, which we prove by induction on the value of the variable i, is
S(j): If we reach the loop test i ≤ n with the variable i having the value j, then the value of the variable fact is (j − 1)!.
STATEMENT
BASIS. The basis is S(2). We reach the test with i having value 2 only when we enter the loop from the outside. Prior to the loop, lines (2) and (3) of Fig. 2.13 set fact to 1 and i to 2. Since 1 = (2 − 1)!, the basis is proved. INDUCTION. Assume S(j), and prove S(j + 1). If j > n, then we break out of the while-loop when i has the value j or earlier, and thus we never reach the loop test with i having the value j + 1. In that case, S(j + 1) is trivially true, because it is of the form “If we reach · · · .” Thus, assume j ≤ n, and consider what happens when we execute the body of the while-loop with i having the value j. By the inductive hypothesis, before line (5) is executed, fact has value (j − 1)!, and i has the value j. Thus, after line (5) is executed, fact has the value j × (j − 1)!, which is j!. At line (6), i is incremented by 1 and so attains the value j + 1. Thus, when we reach the loop test with i having value j + 1, the value of fact is j!. The statement S(j + 1) says that when i equals j + 1, fact equals ((j + 1) − 1)!, or j!. Thus, we have proved statement S(j + 1), and completed the inductive step. We already have shown that the while-loop will terminate. Evidently, it terminates when i first attains a value greater than n. Since i is an integer and is incremented by 1 each time around the loop, i must have the value n + 1 when the loop terminates. Thus, when we reach line (7), statement S(n + 1) must hold. But that statement says that fact has the value n!. Thus, the program prints n!, as we wished to prove.
As a practical matter, we should point out that on any computer the factorial program in Fig. 2.13 will print n! as an answer for very few values of n. The problem is that the factorial function grows so rapidly that the size of the answer quickly exceeds the maximum size of an integer on any real computer. ✦
EXERCISES 2.5.1: What is an appropriate loop invariant for the following program fragment, which sets sum equal to the sum of the integers from 1 to n? scanf("%d",&n); sum = 0; for (i = 1; i <= n; i++) sum = sum + i; Prove your loop invariant by induction on i, and use it to prove that the program works as intended. 2.5.2: The following fragment computes the sum of the integers in array A[0..n-1]:
SEC. 2.6
RECURSIVE DEFINITIONS
59
sum = 0; for (i = 0; i < n; i++) sum = sum + A[i]; What is an appropriate loop invariant? Use it to show that the fragment works as intended. 2.5.3*: Consider the following fragment: scanf("%d", &n); x = 2; for (i = 1; i <= n; i++) x = x * x; An appropriate loop invariant for the point just before the test for i ≤ n is that if k−1 we reach that point with the value k for variable i, then x = 22 . Prove that this invariant holds, by induction on k. What is the value of x after the loop terminates? sum = 0; scanf("%d", &x); while (x >= 0) { sum = sum + x; scanf("%d", &x); } Fig. 2.14. Summing a list of integers terminated by a negative integer.
2.5.4*: The fragment in Fig. 2.14 reads integers until it finds a negative integer, and then prints the accumulated sum. What is an appropriate loop invariant for the point just before the loop test? Use the invariant to show that the fragment performs as intended. 2.5.5: Find the largest value of n for which the program in Fig. 2.13 works on your computer. What are the implications of fixed-length integers for proving programs correct? 2.5.6: Show by induction on the number of times around the loop of Fig. 2.10 that j > i + 1 after the first time around. ✦ ✦ ✦ ✦ Inductive definition
2.6
Recursive Definitions In a recursive, or inductive, definition, we define one or more classes of closely related objects (or facts) in terms of the objects themselves. The definition must not be meaningless, like “a widget is a widget of some color,” or paradoxical, like “something is a glotz if and only if it is not a glotz.” Rather, a recursive definition involves 1.
One or more basis rules, in which some simple objects are defined, and
2.
One or more inductive rules, whereby larger objects are defined in terms of smaller ones in the collection.
60
✦
ITERATION, INDUCTION, AND RECURSION
Example 2.15. In the previous section we defined the factorial function by an iterative algorithm: multiply 1 × 2 × · · · × n to get n!. However, we can also define the value of n! recursively, as follows. BASIS.
1! = 1.
INDUCTION.
n! = n × (n − 1)!.
For example, the basis tells us that 1! = 1. We can use this fact in the inductive step with n = 2 to find 2! = 2 × 1! = 2 × 1 = 2 With n = 3, 4, and 5, we get 3! = 3 × 2! = 3 × 2 = 6 4! = 4 × 3! = 4 × 6 = 24 5! = 5 × 4! = 5 × 24 = 120 and so on. Notice that, although it appears that the term “factorial” is defined in terms of itself, in practice, we can get the value of n! for progressively higher values of n in terms of the factorials for lower values of n only. Thus, we have a meaningful definition of “factorial.” Strictly speaking, we should prove that our recursive definition of n! gives the same result as our original definition, n! = 1 × 2 × · · · × n To do so, we shall prove the following statement:
STATEMENT
S(n): n!, as defined recursively above, equals 1 × 2 × · · · × n.
The proof will be by induction on n. S(1) clearly holds. The basis of the recursive definition tells us that 1! = 1, and the product 1 × · · · × 1 (i.e., the product of the integers “from 1 to 1”) is evidently 1 as well.
BASIS.
INDUCTION. Assume that S(n) holds; that is, n!, as given by the recursive definition, equals 1 × 2 × · · · × n. Then the recursive definition tells us that
(n + 1)! = (n + 1) × n! If we use the commutative law for multiplication, we see that (n + 1)! = n! × (n + 1) By the inductive hypothesis, n! = 1 × 2 × · · · × n Thus, we may substitute 1 × 2 × · · · × n for n! in Equation (2.11) to get (n + 1)! = 1 × 2 × · · · × n × (n + 1)
(2.11)
SEC. 2.6
RECURSIVE DEFINITIONS
61
which is the statement S(n + 1). We have thereby proved the inductive hypothesis and shown that our recursive definition of n! is the same as our iterative definition.✦
nth use of inductive step
···
2nd use of inductive step
1st use of inductive step
basis case
Fig. 2.15. In a recursive definition, we construct objects in rounds, where the objects constructed in one round may depend on objects constructed in all previous rounds.
Figure 2.15 suggests the general nature of a recursive definition. It is similar in structure to a complete induction, in that there is an infinite sequence of cases, each of which can depend on any or all of the previous cases. We start by applying the basis rule or rules. On the next round, we apply the inductive rule or rules to what we have already obtained, to construct new facts or objects. On the following round, we again apply the inductive rules to what we have, obtaining new facts or objects, and so on. In Example 2.15, where we were defining the factorial, we discovered the value of 1! by the basis case, 2! by one application of the inductive step, 3! by two applications, and so on. Here, the induction had the form of an “ordinary” induction, where we used in each round only what we had discovered in the previous round.
✦ Lexicographic order
Example 2.16. In Section 2.2 we defined the notion of lexicographic order of strings, and our definition was iterative in nature. Roughly, we test whether string c1 · · · cn precedes string d1 · · · dm by comparing corresponding symbols ci and di from the left, until we either find an i for which ci 6= di or come to the end of one of the strings. The following recursive definition defines those pairs of strings w and x such that w precedes x in lexicographic order. Intuitively, the induction is on the number of pairs of equal characters at the beginnings of the two strings involved. BASIS. The basis covers those pairs of strings for which we can immediately resolve the question of which comes first in lexicographic order. There are two parts of the basis.
1.
ǫ < w for any string w other than ǫ itself. Recall that ǫ is the empty string, or the string with no characters.
2.
If c < d, where c and d are characters, then for any strings w and x, we have cw < dx.
62
ITERATION, INDUCTION, AND RECURSION
INDUCTION.
If w < x for strings w and x, then for any character c we have
cw < cx. For instance, we can use the above definition to show that base < batter. By rule (2) of the basis, with c = s, d = t, w = e, and x = ter, we have se < tter. If we apply the recursive rule once, with c = a, w = se, and x = tter, we infer that ase < atter. Finally, applying the recursive rule a second time with c = b, w = ase, and x = atter, we find base < batter. That is, the basis and inductive steps appear as follows: se < tter ase < atter base < batter We can also show that bat < batter as follows. Part (1) of the basis tells us that ǫ < ter. If we apply the recursive rule three times — with c equal to t, a, and b, in turn — we make the following sequence of inferences: ǫ < ter t < tter at < atter bat < batter Now we should prove, by induction on the number of characters that two strings have in common at their left ends, that one string precedes the other according to the definition in Section 2.2 if and only if it precedes according to the recursive definition just given. We leave these two inductive proofs as exercises. ✦ In Example 2.16, the groups of facts suggested by Fig. 2.15 are large. The basis case gives us all facts w < x for which either w = ǫ or w and x begin with different letters. One use of the inductive step gives us all w < x facts where w and x have exactly one initial letter in common, the second use covers those cases where w and x have exactly two initial letters in common, and so on.
Expressions Arithmetic expressions of all kinds are naturally defined recursively. For the basis of the definition, we specify what the atomic operands can be. For example, in C, atomic operands are either variables or constants. Then, the induction tells us what operators may be applied, and to how many operands each is applied. For instance, in C, the operator < can be applied to two operands, the operator symbol − can be applied to one or two operands, and the function application operator, represented by a pair of parenthesis with as many commas inside as necessary, can be applied to one or more operands, as f (a1 , . . . , an ).
✦
Example 2.17. It is common to refer to the following set of expressions as “arithmetic expressions.” BASIS.
The following types of atomic operands are arithmetic expressions:
SEC. 2.6
1. 2. 3.
RECURSIVE DEFINITIONS
63
Variables Integers Real numbers
If E1 and E2 are arithmetic expressions, then the following are also arithmetic expressions:
INDUCTION.
1. 2. 3. 4. Infix operator
(E1 + E2 ) (E1 − E2 ) (E1 × E2 ) (E1 / E2 )
The operators +, −, ×, and / are said to be binary operators, because they take two arguments. They are also said to be infix operators, because they appear between their two arguments. Additionally, we allow a minus sign to imply negation (change of sign), as well as subtraction. That possibility is reflected in the fifth and last recursive rule: 5.
Unary, prefix operator
If E is an arithmetic expression, then so is (−E).
An operator like − in rule (5), which takes only one operand, is said to be a unary operator. It is also said to be a prefix operator, because it appears before its argument. Figure 2.16 illustrates some arithmetic expressions and explains why each is an expression. Note that sometimes parentheses are not needed, and we can omit them. In the final expression (vi) of Fig. 2.16, the outer parentheses and the parentheses around −(x + 10) can be omitted, and we could write y × −(x + 10). However, the remaining parentheses are essential, since y × −x + 10 is conventionally interpreted as (y × −x) + 10, which is not an equivalent expression (try y = 1 and x = 0, for instance).7 ✦ i) ii) iii) iv) v) vi)
x 10 (x + 10) −(x + 10) y
y × −(x + 10)
Basis rule (1) Basis rule (2) Recursive rule (1) on (i) and (ii) Recursive rule (5) on (iii) Basis rule (1) Recursive rule (3) on (v) and (iv)
Fig. 2.16. Some sample arithmetic expressions.
7
Parentheses are redundant when they are implied by the conventional precedences of operators (unary minus highest, then multiplication and division, then addition and subtraction) and by the convention of “left associativity,” which says that we group operators at the same precedence level (e.g., a string of plusses and minuses) from the left. These conventions should be familiar from C, as well as from ordinary arithmetic.
64
ITERATION, INDUCTION, AND RECURSION
More Operator Terminology Postfix operator
Ternary operator
A unary operator that appears after its argument, as does the factorial operator ! in expressions like n!, is said to be a postfix operator. Operators that take more than one operand can also be prefix or postfix operators, if they appear before or after all their arguments, respectively. There are no examples in C or ordinary arithmetic of operators of these types, although in Section 5.4 we shall discuss notations in which all operators are prefix or postfix operators. An operator that takes three arguments is a ternary operator. In C, ?: is a ternary operator, as in the expression c?x:y meaning “if c then x else y.” In general, if an operator takes k arguments, it is said to be k-ary.
Balanced Parentheses Strings of parentheses that can appear in expressions are called balanced parentheses. For example, the pattern ((())) appears in expression (vi) of Fig. 2.16, and the expression (a + b) × (c + d) − e has the pattern (()(())). The empty string, ǫ, is also a string of balanced parentheses; it is the pattern of the expression x, for example. In general, what makes a string of parentheses balanced is that it is possible to match each left parenthesis with a right parenthesis that appears somewhere to its right. Thus, a common definition of “balanced parenthesis strings” consists of two rules:
Profile
1.
A balanced string has an equal number of left and right parentheses.
2.
As we move from left to right along the string, the profile of the string never becomes negative, where the profile is the running total of the number of left parentheses seen minus the number of right parentheses seen.
Note that the profile must begin and end at 0. For example, Fig. 2.17(a) shows the profile of (()(())), and Fig. 2.17(b) shows the profile of ()(())(). There are a number of recursive definitions for the notion of “balanced parentheses.” The following is a bit subtle, but we shall prove that it is equivalent to the preceding, nonrecursive definition involving profiles. BASIS.
The empty string is a string of balanced parentheses.
INDUCTION. If x and y are strings of balanced parentheses, then (x)y is also a string of balanced parentheses.
✦
Example 2.18. By the basis, ǫ is a balanced-parenthesis string. If we apply the recursive rule, with x and y both equal to ǫ, then we infer that () is balanced. Notice that when we substitute the empty string for a variable, such as x or y, that variable “disappears.” Then we may apply the recursive rule with:
SEC. 2.6
RECURSIVE DEFINITIONS
65
3 2 1 0
(
(
)
(
(
)
)
)
(
)
(a) Profile of (()(())).
2 1 0 (
)
(
(
)
)
(b) Profile of ()(())(). Fig. 2.17. Profiles of two strings of parentheses.
1.
x = () and y = ǫ, to discover that (()) is balanced.
2.
x = ǫ and y = (), to find that ()() is balanced.
3.
x = y = () to infer that (())() is balanced.
As a final example, since we now know that (()) and ()() are balanced, we may let these be x and y in the recursive rule, respectively, and show that ((()))()() is balanced. ✦
Profile-balanced
We can show that the two definitions of “balanced” specify the same sets of strings. To make things clearer, let us refer to strings that are balanced according to the recursive definition simply as balanced and refer to those balanced according to the nonrecursive definition as profile-balanced. That is, the profile-balanced strings are those whose profile ends at 0 and never goes negative. We need to show two things: 1.
Every balanced string is profile-balanced.
2.
Every profile-balanced string is balanced.
These are the aims of the inductive proofs in the next two examples.
✦
Example 2.19. First, let us prove part (1), that every balanced string is profilebalanced. The proof is a complete induction that mirrors the induction by which the class of balanced strings is defined. That is, we prove
S(n): If string w is defined to be balanced by n applications of the recursive rule, then w is profile-balanced.
STATEMENT
66
ITERATION, INDUCTION, AND RECURSION
BASIS. The basis is n = 0. The only string that can be shown to be balanced without any application of the recursive rule is ǫ, which is balanced according to the basis rule. Evidently, the profile of the empty string ends at 0 and does not go negative, so ǫ is profile-balanced.
Assume that S(i) is true for i = 0, 1, . . . , n, and consider an instance of S(n + 1), that is, a string w whose proof of balance requires n + 1 uses of the recursive rule. Consider the last such use, in which we took two strings x and y, already known to be balanced, and formed w as (x)y. We used the recursive rule n + 1 times to form w, and one use was the last step, which helped form neither x nor y. Thus, neither x nor y requires more than n uses of the recursive rule. Therefore, the inductive hypothesis applies to both x and y, and we can conclude that x and y are profile-balanced. INDUCTION.
profile of x
profile of y
1 0
(
x
)
y
Fig. 2.18. Constructing the profile of w = (x)y.
The profile of w is as suggested in Fig. 2.18. It first goes up one level, in response to the first left parenthesis. Then comes the profile of x, raised one level, as indicated by the dashed line. We used the inductive hypothesis to conclude that x is profile-balanced; therefore, its profile begins and ends at level 0 and never goes negative. As the x portion of w’s profile is raised one level in Fig. 2.18, that portion begins and ends at level 1 and never goes below level 1. The explicitly shown right parenthesis between x and y lowers the profile of w to 0. Then comes the profile of y. By the inductive hypothesis, y is profile-balanced. Thus, the y portion of w’s profile does not go below 0, and it ends the profile of w at 0. We have now constructed the profile of w and see that it meets the condition for a profile-balanced string. That is, it begins and ends at 0, and it never becomes negative. Thus, we have proved that if a string is balanced, it is profile-balanced. ✦ Now we shall address the second direction of the equivalence between the two definitions of “balanced parentheses.” We show in the next example that a profilebalanced string is balanced.
✦
Example 2.20. We prove part (2), that “profile-balanced” implies “balanced,” by complete induction on the length of the string of parentheses. The formal statement is
SEC. 2.6
RECURSIVE DEFINITIONS
67
Proofs About Recursive Definitions Notice that Example 2.19 proves an assertion about a class of recursively defined objects (the balanced strings of parentheses) by induction on the number of times the recursive rule is used to establish that the object is in the defined class. That is a very common way to deal with recursively defined concepts; in fact, it is one of the reasons recursive definitions are useful. As another illustration, in Example 2.15, we showed a property of the recursively defined factorial values (that n! is the product of the integers from 1 to n) by induction on n. But n is also 1 plus the number of times we used the recursive rule in the definition of n!, so the proof could also be considered an induction on the number of applications of the recursive rule.
STATEMENT
S(n): If a string w of length n is profile-balanced, then it is balanced.
If n = 0, then the string must be ǫ. We know that ǫ is balanced by the basis rule of the recursive definition.
BASIS.
Suppose that profile-balanced strings of length equal to or less than n are balanced. We must prove S(n + 1), that profile-balanced strings of length n + 1 are also balanced.8 Consider such a string w. Since w is profile-balanced, it cannot start with a right parenthesis, or its profile would immediately go negative. Thus, w begins with a left parenthesis. Let us break w into two parts. The first part starts at the beginning of w and ends where the profile of w first becomes 0. The second part is the remainder of w. For example, the profile of Fig. 2.17(a) first becomes 0 at the end, so if w = (()(())), then the first part is the entire string and the second part is ǫ. In Fig. 2.17(b), where w = ()(())(), the first part is (), and the second part is (())(). The first part can never end in a left parenthesis, because then the profile would be negative at the position just before. Thus, the first part begins with a left parenthesis and ends with a right parenthesis. We can therefore write w as (x)y, where (x) is the first part and y is the second part. Both x and y are shorter than w, so if we can show they are profile-balanced, then we can use the inductive hypothesis to infer that they are balanced. Then we can use the recursive rule in the definition of “balanced” to show that w = (x)y is balanced. It is easy to see that y is profile-balanced. Figure 2.18 also illustrates the relationship between the profiles of w, x, and y here. That is, the profile of y is a tail of the profile of w, beginning and ending at height 0. Since w is profilebalanced, we can conclude that y is also. Showing that x is profile-balanced is almost the same. The profile of x is a part of the profile of w; it begins and ends at level 1 in the profile of w, but we can lower it by one level to get the profile of x. We know that the profile of w never reaches 0 during the extent of x, because we picked (x) to be the shortest prefix of w that ends with the profile of w at level 0. Hence, the profile of x within w never reaches level 0, and the profile of x itself never becomes negative. We have now shown both x and y to be profile-balanced. Since they are each
INDUCTION.
8
Note that all profile-balanced strings happen to be of even length, so if n + 1 is odd, we are not saying anything. However, we do not need the evenness of n for the proof.
68
ITERATION, INDUCTION, AND RECURSION
shorter than w, the inductive hypothesis applies to them, and they are each balanced. The recursive rule defining “balanced” says that if x and y are balanced, then so is (x)y. But w = (x)y, and so w is balanced. We have now completed the inductive step and shown statement S(n) to be true for all n ≥ 0. ✦
EXERCISES 2.6.1*: Prove that the definitions of lexicographic order given in Example 2.16 and in Section 2.2 are the same. Hint : The proof consists of two parts, and each is an inductive proof. For the first part, suppose that w < x according to the definition in Example 2.16. Prove the following statement S(i) by induction on i: “If it is necessary to apply the recursive rule i times to show that w < x, then w precedes x according to the definition of ‘lexicographic order’ in Section 2.2.” The basis is i = 0. The second part of the exercise is to show that if w precedes x in lexicographic order according to the definition in Section 2.2, then w < x according to the definition in Example 2.16. Now the induction is on the number of initial positions that w and x have in common. 2.6.2: Draw the profiles of the following strings of parentheses: a) b) c) d)
(()(()) ()())(() ((()())()()) (()(()(())))
Which are profile-balanced? For those that are profile-balanced, use the recursive definition in Section 2.6 to show that they are balanced. 2.6.3*: Show that every string of balanced parentheses (according to the recursive definition in Section 2.6) is the string of parentheses in some arithmetic expression (see Example 2.17 for a definition of arithmetic expressions). Hint : Use a proof by induction on the number of times the recursive rule of the definition of “balanced parentheses” is used to construct the given string of balanced parentheses. 2.6.4: Tell whether each of the following C operators is prefix, postfix, or infix, and whether they are unary, binary, or k-ary for some k > 2: a) < b) & c) % 2.6.5: If you are familiar with the UNIX file system or a similar system, give a recursive definition of the possible directory/file structures. 2.6.6*: A certain set S of integers is defined recursively by the following rules. BASIS.
0 is in S.
INDUCTION.
a)
If i is in S, then i + 5 and i + 7 are in S.
What is the largest integer not in S?
SEC. 2.7
b)
RECURSIVE FUNCTIONS
69
Let j be your answer to part (a). Prove that all integers j + 1 and greater are in S. Hint : Note the similarity to Exercise 2.4.8 (although here we are dealing with only nonnegative integers).
2.6.7*: Define recursively the set of even-parity strings, by induction on the length of the string. Hint : It helps to define two concepts simultaneously, both the evenparity strings and the odd-parity strings. 2.6.8*: We can define sorted lists of integers as follows. BASIS.
A list consisting of a single integer is sorted.
INDUCTION. If L is a sorted list in which the last element is a, and if b ≥ a, then L followed by b is a sorted list.
Prove that this recursive definition of “sorted list” is equivalent to our original, nonrecursive definition, which is that the list consist of integers a1 ≤ a2 ≤ · · · ≤ an Remember, you need to prove two parts: (a) If a list is sorted by the recursive definition, then it is sorted by the nonrecursive definition, and (b) if a list is sorted by the nonrecursive definition, then it is sorted by the recursive definition. Part (a) can use induction on the number of times the recursive rule is used, and (b) can use induction on the length of the list. 2.6.9**: As suggested by Fig. 2.15, whenever we have a recursive definition, we can classify the objects defined according to the “round” on which each is generated, that is, the number of times the inductive step is applied before we obtain each object. In Examples 2.15 and 2.16, it was fairly easy to describe the results generated on each round. Sometimes it is more challenging to do so. How do you characterize the objects generated on the nth round for each of the following?
✦ ✦ ✦ ✦ Direct and indirect recursion
2.7
a)
Arithmetic expressions like those described in Example 2.17. Hint : If you are familiar with trees, which are the subject of Chapter 5, you might consider the tree representation of expressions.
b)
Balanced parenthesis strings. Note that the “number of applications used,” as discussed in Example 2.19, is not the same as the round on which a string is discovered. For example, (())() uses the inductive rule three times but is discovered on round 2.
Recursive Functions A recursive function is one that is called from within its own body. Often, the call is direct; for example, a function F has a call to F within itself. Sometimes, however, the call is indirect: some function F1 calls a function F2 directly, which calls F3 directly, and so on, until some function Fk in the sequence calls F1 . There is a common belief that it is easier to learn to program iteratively, or to use nonrecursive function calls, than it is to learn to program recursively. While we cannot argue conclusively against that point of view, we do believe that recursive programming is easy once one has had the opportunity to practice the style.
70
ITERATION, INDUCTION, AND RECURSION
More Truth in Advertising A potential disadvantage of using recursion is that function calls on some machines are time-consuming, so that a recursive program may take more time to run than an iterative program for the same problem. However, on many modern machines function calls are quite efficient, and so this argument against using recursive programs is becoming less important. Even on machines with slow function-calling mechanisms, one can profile a program to find how much time is spent on each part of the program. One can then rewrite the parts of the program in which the bulk of its time is spent, replacing recursion by iteration if necessary. That way, one gets the advantages of recursion throughout most of the program, except for a small fraction of the code where speed is most critical.
Profiling
Recursive programs are often more succinct or easier to understand than their iterative counterparts. More importantly, some problems are more easily attacked by recursive programs than by iterative programs.9 Often, we can develop a recursive algorithm by mimicking a recursive definition in the specification of a program we are trying to implement. A recursive function that implements a recursive definition will have a basis part and an inductive part. Frequently, the basis part checks for a simple kind of input that can be solved by the basis of the definition, with no recursive call needed. The inductive part of the function requires one or more recursive calls to itself and implements the inductive part of the definition. Some examples should clarify these points.
✦
Example 2.21. Figure 2.19 gives a recursive function that computes n! given a positive integer n. This function is a direct transcription of the recursive definition of n! in Example 2.15. That is, line (1) of Fig. 2.19 distinguishes the basis case from the inductive case. We assume that n ≥ 1, so the test of line (1) is really asking whether n = 1. If so, we apply the basis rule, 1! = 1, at line (2). If n > 1, then we apply the inductive rule, n! = n × (n − 1)!, at line (3).
(1) (2) (3)
int fact(int n) { if (n <= 1) return 1; /* basis */ else return n*fact(n-1); /* induction */ } Fig. 2.19. Recursive function to compute n! for n ≥ 1.
For instance, if we call fact(4), the result is a call to fact(3), which calls 9
Such problems often involve some kind of search. For instance, in Chapter 5 we shall see some recursive algorithms for searching trees, algorithms that have no convenient iterative analog (although there are equivalent iterative algorithms using stacks).
SEC. 2.7
RECURSIVE FUNCTIONS
71
Defensive Programming The program of Fig. 2.19 illustrates an important point about writing recursive programs so that they do not run off into infinite sequences of calls. We tacitly assumed that fact would never be called with an argument less than 1. Best, of course, is to begin fact with a test that n ≥ 1, printing an error message and returning some particular value such as 0 if it is not. However, even if we believe very strongly that fact will never be called with n < 1, we shall be wise to include in the basis case all these “error cases.” Then, the function fact called with erroneous input will simply return the value 1, which is wrong, but not a disaster (in fact, 1 is even correct for n = 0, since 0! is conventionally defined to be 1). However, suppose we were to ignore the error cases and write line (1) of Fig. 2.19 as if (n == 1) Then if we called fact(0), it would look like an instance of the inductive case, and we would next call fact(-1), then fact(-2), and so on, terminating with failure when the computer ran out of space to record the recursive calls. fact(2), which calls fact(1). At that point, fact(1) applies the basis rule, because n ≤ 1, and returns the value 1 to fact(2). That call to fact completes line (3), returning 2 to fact(3). In turn, fact(3) returns 6 to fact(4), which completes line (3) by returning 24 as the answer. Figure 2.20 suggests the pattern of calls and returns. ✦
Call ↓ ↑ Return 24 fact(4) fact(4) Call ↓ ↑ Return 6 fact(3) fact(3) Call ↓ ↑ Return 2 fact(2) fact(2) Call ↓ ↑ Return 1 fact(1) Fig. 2.20. Calls and returns resulting from call to fact(4).
Size of arguments
We can picture a recursion much as we have pictured inductive proofs and definitions. In Fig. 2.21 we have assumed that there is a notion of the “size” of arguments for a recursive function. For example, for the function fact in Example 2.21 the value of the argument n itself is the appropriate size. We shall say more about the matter of size in Section 2.9. However, let us note here that it is essential for a recursion to make only calls involving arguments of smaller size. Also, we must reach the basis case — that is, we must terminate the recursion — when we reach some particular size, which in Fig. 2.21 is size 0. In the case of the function fact, the calls are not as general as suggested by Fig. 2.21. A call to fact(n) results in a direct call to fact(n-1), but fact(n) does
72
ITERATION, INDUCTION, AND RECURSION
size n
···
size 2
size 1
size 0
Fig. 2.21. A recursive function calls itself with arguments of smaller size.
not call fact with any smaller argument directly.
✦
Example 2.22. We can turn the function SelectionSort of Fig. 2.2 into a recursive function recSS, if we express the underlying algorithm as follows. Assume the data to be sorted is in A[0..n-1]. 1.
Pick a smallest element from the tail of the array A, that is, from A[i..n-1].
2.
Swap the element selected in step (1) with A[i].
3.
Sort the remainder of the array, A[i+1..n-1].
We can express selection sort as the following recursive algorithm. BASIS. If i = n − 1, then only the last element of the array remains to be sorted. Since any one element is already sorted, we need not do anything.
If i < n − 1, then find the smallest element in A[i..n-1], swap it with A[i], and recursively sort A[i+1..n-1].
INDUCTION.
Backward induction
The entire algorithm is to perform the above recursion starting with i = 0. If we see i as the parameter in the preceding induction, it is a case of backward induction, where we start with a high basis and by the inductive rule solve instances with smaller values of the parameter in terms of instances with higher values. That is a perfectly good style of induction, although we have not previously mentioned its possibility. However, we can also see this induction as an ordinary, or “forward” induction on a parameter k = n − i that represents the number of elements in the tail of the array waiting to be sorted. In Fig. 2.22, we see the program for recSS(A,i,n). The second parameter i is the index of the first element in the unsorted tail of the array A. The third parameter n is the total number of elements in the array A to be sorted. Presumably, n is less than or equal to the maximum size of A. Thus, a call to recSS(A,0,n) will sort the entire array A[0..n-1]. In terms of Fig. 2.21, s = n− i is the appropriate notion of “size” for arguments of the function recSS. The basis is s = 1 — that is, sorting one element, in which case no recursive calls occur. The inductive step tells us how to sort s elements by picking the smallest and then sorting the remaining s − 1 elements. At line (1) we test for the basis case, in which there is only one element remaining to be sorted (again, we are being defensive, so that if we somehow make a
SEC. 2.7
(1)
(2) (3) (4) (5) (6) (7) (8) (9)
RECURSIVE FUNCTIONS
73
void recSS(int A[], int i, int n) { int j, small, temp; if (i < n-1) {/* basis is when i = n-1, in which case */ /* the function returns without changing A */ /* induction follows */ small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j; temp = A[small]; A[small] = A[i]; A[i] = temp; recSS(A, i+1, n); } } Fig. 2.22. Recursive selection sort.
call with i ≥ n, we shall not go into an infinite sequence of calls). In the basis case, we have nothing to do, so we just return. The remainder of the function is the inductive case. Lines (2) through (8) are copied directly from the iterative version of selection sort. Like that program, these lines set small to the index of the array A[i..n-1] that holds a smallest element and then swap this element with A[i]. Finally, line (9) is the recursive call, which sorts the remainder of the array. ✦
EXERCISES 2.7.1: We can define n2 recursively as follows. BASIS.
For n = 1, 12 = 1.
INDUCTION.
If n2 = m, then (n + 1)2 = m + 2n + 1.
a)
Write a recursive C function to implement this recursion.
b)
Prove by induction on n that this definition correctly computes n2 .
2.7.2: Suppose that we are given array A[0..4], with elements 10, 13, 4, 7, 11, in that order. What are the contents of the array A just before each recursive call to recSS, according to the recursive function of Fig. 2.22? 2.7.3: Suppose we define cells for a linked list of integers, as discussed in Section 1.3, using the macro DefCell(int, CELL, LIST) of Section 1.6. Recall, this macro expands to be the following type definition:
74
ITERATION, INDUCTION, AND RECURSION
Divide-and-Conquer One way of attacking a problem is to try to break it into subproblems and then solve the subproblems and combine their solutions into a solution for the problem as a whole. The term divide-and-conquer is used to describe this problem-solving technique. If the subproblems are similar to the original, then we may be able to use the same function to solve the subproblems recursively. There are two requirements for this technique to work. The first is that the subproblems must be simpler than the original problem. The second is that after a finite number of subdivisions, we must encounter a subproblem that can be solved outright. If these criteria are not met, a recursive algorithm will continue subdividing the problem forever, without finding a solution. Note that the recursive function recSS in Fig. 2.22 satisfies both criteria. Each time it is invoked, it is on a subarray that has one fewer element, and when it is invoked on a subarray containing a single element, it returns without invoking itself again. Similarly, the factorial program of Fig. 2.19 involves calls with a smaller integer value at each call, and the recursion stops when the argument of the call reaches 1. Section 2.8 discusses a more powerful use of the divide-and-conquer technique, called “merge sort.” There, the size of the arrays being sorted diminishes very rapidly, because merge sort works by dividing the size in half, rather than subtracting 1, at each recursive call.
typedef struct CELL *LIST; struct CELL { int element; LIST next; }; Write a recursive function find that takes an argument of type LIST and returns TRUE if some cell of the list contains the integer 1698 as its element and returns FALSE if not. 2.7.4: Write a recursive function add that takes an argument of type LIST, as defined in Exercise 2.7.3, and returns the sum of the elements on the list. 2.7.5: Write a version of recursive selection sort that takes as argument a list of integers, using the cells mentioned in Exercise 2.7.3. 2.7.6: In Exercise 2.2.8 we suggested that one could generalize selection sort to use arbitrary key and lt functions to compare elements. Rewrite the recursive selection sort algorithm to incorporate this generality. 2.7.7*: Give a recursive algorithm that takes an integer i and produces the binary representation of i as a sequence of 0’s and 1’s, low-order bit first. GCD
2.7.8*: The greatest common divisor (GCD) of two integers i and j is the largest integer that divides both i and j evenly. For example, gcd(24, 30) = 6, and gcd(24, 35) = 1. Write a recursive function that takes two integers i and j, with i > j, and returns gcd(i, j). Hint : You may use the following recursive definition of gcd. It assumes that i > j.
SEC. 2.8
BASIS.
MERGE SORT: A RECURSIVE SORTING ALGORITHM
75
If j divides i evenly, then j is the GCD of i and j.
INDUCTION. If j does not divide i evenly, let k be the remainder when i is divided by j. Then gcd(i, j) is the same as gcd(j, k).
2.7.9**: Prove that the recursive definition of GCD given in Exercise 2.7.8 gives the same result as the nonrecursive definition (largest integer dividing both i and j evenly). 2.7.10: Often, a recursive definition can be turned into an algorithm fairly directly. For example, consider the recursive definition of “less than” on strings given in Example 2.16. Write a recursive function that tests whether the first of two given strings is “less than” the other. Assume that strings are represented by linked lists of characters. 2.7.11*: From the recursive definition of a sorted list given in Exercise 2.6.8, create a recursive sorting algorithm. How does this algorithm compare with the recursive selection sort of Example 2.22? ✦ ✦ ✦ ✦
Divide and conquer
2.8
Merge Sort: A Recursive Sorting Algorithm We shall now consider a sorting algorithm, called merge sort, which is radically different from selection sort. Merge sort is best described recursively, and it illustrates a powerful use of the divide-and-conquer technique, in which we sort a list (a1 , a2 , . . . , an ) by “dividing” the problem into two similar problems of half the size. In principle, we could begin by dividing the list into two arbitrarily chosen equal-sized lists, but in the program we develop, we shall make one list out of the odd-numbered elements, (a1 , a3 , a5 , . . .) and the other out of the even-numbered elements, (a2 , a4 , a6 , . . .).10 We then sort each of the half-sized lists separately. To complete the sorting of the original list of n elements, we merge the two sorted, half-sized lists by an algorithm to be described in the next example. In the next chapter, we shall see that the time required for merge sort grows much more slowly, as a function of the length n of the list to be sorted, than does the time required by selection sort. Thus, even if recursive calls take some extra time, merge sort is greatly preferable to selection sort when n is large. In Chapter 3 we shall examine the relative performance of these two sorting algorithms.
Merging To “merge” means to produce from two sorted lists a single sorted list containing all the elements of the two given lists and no other elements. For example, given the lists (1, 2, 7, 7, 9) and (2, 4, 7, 8), the merger of these lists is (1, 2, 2, 4, 7, 7, 7, 8, 9). Note that it does not make sense to talk about “merging” lists that are not already sorted. One simple way to merge two lists is to examine them from the front. At each step, we find the smaller of the two elements at the current fronts of the lists, choose that element as the next element on the combined list, and remove the chosen element from its list, exposing a new “first” element on that list. Ties can be broken 10
Remember that “odd-numbered” and “even-numbered” refer to the positions of the elements on the list, and not to the values of these elements.
76
ITERATION, INDUCTION, AND RECURSION
L1
L2
1, 2, 7, 7, 9 2, 7, 7, 9 7, 7, 9 7, 7, 9 7, 7, 9 7, 9 9 9 9 empty
2, 4, 7, 8 2, 4, 7, 8 2, 4, 7, 8 4, 7, 8 7, 8 7, 8 7, 8 8 empty empty
M empty 1 1, 2 1, 2, 2 1, 2, 2, 4 1, 2, 2, 4, 7 1, 2, 2, 4, 7, 7 1, 2, 2, 4, 7, 7, 7 1, 2, 2, 4, 7, 7, 7, 8 1, 2, 2, 4, 7, 7, 7, 8, 9
Fig. 2.23. Example of merging.
arbitrarily, although we shall take from the first list when the leading elements of both lists are the same.
✦
Example 2.23. Consider merging the two lists L1 = (1, 2, 7, 7, 9) and L2 = (2, 4, 7, 8) The first elements of the lists are 1 and 2, respectively. Since 1 is smaller, we choose it as the first element of the merged list M and remove 1 from L1 . The new L1 is thus (2, 7, 7, 9). Now, both L1 and L2 have 2 as their first elements. We can pick either. Suppose we adopt the policy that we always pick the element from L1 in case of a tie. Then merged list M becomes (1, 2), list L1 becomes (7, 7, 9), and L2 remains (2, 4, 7, 8). The table in Fig. 2.23 shows the merging steps until lists L1 and L2 are both exhausted. ✦ We shall find it easier to design a recursive merging algorithm if we represent lists in the linked form suggested in Section 1.3. Linked lists will be reviewed in more detail in Chapter 6. In what follows, we shall assume that list elements are integers. Thus, each element can be represented by a “cell,” or structure of the type struct CELL, and the list by a type LIST, which is a pointer to a CELL. These definitions are provided by the macro DefCell(int, CELL, LIST), which we discussed in Section 1.6. This use of macro DefCell expands into: typedef struct CELL *LIST; struct CELL { int element; LIST next; }; The element field of each cell contains an integer, and the next field contains a pointer to the next cell on the list. If the element at hand is the last on the list, then the next field contains the value NULL, which represents a null pointer. A list of integers is then represented by a pointer to the first cell on the list, that is, by a variable of type LIST. An empty list is represented by a variable with the value
SEC. 2.8
(1) (2) (3)
(4) (5)
(6) (7)
MERGE SORT: A RECURSIVE SORTING ALGORITHM
77
LIST merge(LIST list1, LIST list2) { if (list1 == NULL) return list2; else if (list2 == NULL) return list1; else if (list1->element <= list2->element) { /* Here, neither list is empty, and the first list has the smaller first element. The answer is the first element of the first list followed by the merge of the remaining elements. */ list1->next = merge(list1->next, list2); return list1; } else { /* list2 has smaller first element */ list2->next = merge(list1, list2->next); return list2; } } Fig. 2.24. Recursive merge.
NULL, in place of a pointer to the first element. Figure 2.24 is a C implementation of a recursive merging algorithm. The function merge takes two lists as arguments and returns the merged list. That is, the formal parameters list1 and list2 are pointers to the two given lists, and the return value is a pointer to the merged list. The recursive algorithm can be described as follows. If either list is empty, then the other list is the desired result. This rule is implemented by lines (1) and (2) of Fig. 2.24. Note that if both lists are empty, then list2 will be returned. But that is correct, since the value of list2 is then NULL and the merger of two empty lists is an empty list.
BASIS.
INDUCTION. If neither list is empty, then each has a first element. We can refer to the two first elements as list1->element and list2->element, that is, the element fields of the cells pointed to by list1 and list2, respectively. Fig 2.25 is a picture of the data structure. The list to be returned begins with the cell of the smallest element. The remainder of the list is formed by merging all but that element. For example, lines (4) and (5) handle the case in which the first element of list 1 is smallest. Line (4) is a recursive call to merge. The first argument of this call is list1->next, that is, a pointer to the second element on the first list (or NULL if the first list only has one element). Thus, the recursive call is passed the list consisting of all but the first element of the first list. The second argument is the entire second list. As a consequence, the recursive call to merge at line (4) will return a pointer to the merged list of all the remaining elements and store a pointer to this merged list in the next field of the first cell on list 1. At line (5), we return a pointer to that cell, which is now the first cell on the merged list of all the elements. Figure 2.25 illustrates the changes. Dotted arrows are present when merge is
78
ITERATION, INDUCTION, AND RECURSION
return value
•
list1
•
to merged list
•
•
··· merged in recursive call
list2
•
•
•
···
Fig. 2.25. Inductive step of merging algorithm.
called. Solid arrows are created by merge. Specifically, the return value of merge is a pointer to the cell of the smallest element, and the next field of that element is shown pointing to the list returned by the recursive call to merge at line (4). Finally, lines (6) and (7) handle the case where the second list has the smallest element. The behavior of the algorithm is exactly as in lines (4) and (5), but the roles of the two lists are reversed.
✦
Example 2.24. Suppose we call merge on the lists (1, 2, 7, 7, 9) and (2, 4, 7, 8) of Example 2.23. Figure 2.26 illustrates the sequence of calls made to merge, if we read the first column downward. We omit the commas separating list elements, but commas are used to separate the arguments of merge. CALL merge(12779, 2478) merge(2779, 2478) merge(779, 2478) merge(779, 478) merge(779, 78) merge(79, 78) merge(9, 78) merge(9, 8) merge(9, NULL)
RETURN 122477789 22477789 2477789 477789 77789 7789 789 89 9
Fig. 2.26. Recursive calls to merge.
For instance, since the first element of list 1 is less than the first element of list 2, line (4) of Fig. 2.24 is executed and we recursively merge all but the first element of list 1. That is, the first argument is the tail of list 1, or (2, 7, 7, 9), and the second argument is the full list 2, or (2, 4, 7, 8). Now the leading elements of both lists are the same. Since the test of line (3) in Fig. 2.24 favors the first list, we remove the 2 from list 1, and our next call to merge has first argument (7, 7, 9) and second argument (2, 4, 7, 8). The returned lists are indicated in the second column, read upward. Notice that, unlike the iterative description of merging suggested by Fig. 2.23, the recursive
SEC. 2.8
MERGE SORT: A RECURSIVE SORTING ALGORITHM
79
algorithm assembles the merged list from the rear, whereas the iterative algorithm assembles it from the front. ✦
Splitting Lists Another important task required for merge sort is splitting a list into two equal parts, or into parts whose lengths differ by 1 if the original list is of odd length. One way to do this job is to count the number of elements on the list, divide by 2, and break the list at the midpoint. Instead, we shall give a simple recursive function split that “deals” the elements into two lists, one consisting of the first, third, and fifth elements, and so on, and the other consisting of the elements at the even positions. More precisely, the function split removes the even-numbered elements from the list it is given as an argument and returns a new list consisting of the even-numbered elements. The C code for function split is shown in Fig. 2.27. Its argument is a list of the type LIST that was defined in connection with the merge function. Note that the local variable pSecondCell is defined to be of type LIST. We really use pSecondCell as a pointer to the second cell on a list, rather than as a list itself; but of course type LIST is, in fact, a pointer to a cell. It is important to observe that split is a function with a side effect. It removes the cells in the even positions from the list it is given as an argument, and it assembles these cells into a new list, which becomes the return value of the function. LIST split(LIST list) { LIST pSecondCell; (1) (2)
if (list == NULL) return NULL; else if (list->next == NULL) return NULL; else { /* there are at least two cells */ pSecondCell = list->next; list->next = pSecondCell->next; pSecondCell->next = split(pSecondCell->next); return pSecondCell; }
(3) (4) (5) (6) }
Fig. 2.27. Splitting a list into two equal pieces.
The splitting algorithm can be described inductively, as follows. It uses an induction on the length of a list; that induction has a multiple basis case. If the list is of length 0 or 1, then we do nothing. That is, an empty list is “split” into two empty lists, and a list of a single element is split by leaving the element on the given list and returning an empty list of the even-numbered elements, of which there are none. The basis is handled by lines (1) and (2) of Fig. 2.27. Line (1) handles the case where list is empty, and line (2) handles the case where it is a single element. Notice that we are careful not to examine list->next in line (2) unless we have previously determined, at line (1), that list is not NULL.
BASIS.
80
ITERATION, INDUCTION, AND RECURSION
INDUCTION. The inductive step applies when there are at least two elements on list. At line (3), we keep a pointer to the second cell of the list in the local variable pSecondCell. Line (4) makes the next field of the first cell skip over the second cell and point to the third cell (or become NULL if there are only two cells on the list). At line (5), we call split recursively, on the list consisting of all but the first two elements. The value returned by that call is a pointer to the fourth element (or NULL if the list is shorter than four elements), and we place this pointer in the next field of the second cell, to complete the linking of the even-numbered elements. A pointer to the second cell is returned by split at line (6); that pointer gives us access to the linked list of all the even-numbered elements of the original list. The changes made by split are suggested in Fig. 2.28. Original pointers are dotted, and new pointers are solid. We also indicate the number of the line that creates each of the new pointers.
(4) list
•
•
•
•
•
(5) (3) pSecondCell
recursive call to split
•
(6) return value
• Fig. 2.28. Action of function split.
The Sorting Algorithm The recursive sorting algorithm is shown in Fig. 2.29. The algorithm can be described by the following basis and inductive step. If the list to be sorted is empty or of length 1, just return the list; it is already sorted. The basis is taken care of by lines (1) and (2) of Fig. 2.29.
BASIS.
If the list is of length at least 2, use the function split at line (3) to remove the even-numbered elements from list and use them to form another list, pointed to by local variable SecondList. Line (4) recursively sorts the half-sized lists, and returns the merger of the two lists. INDUCTION.
SEC. 2.8
MERGE SORT: A RECURSIVE SORTING ALGORITHM
81
LIST MergeSort(LIST list) { LIST SecondList; (1) (2)
if (list == NULL) return NULL; else if (list->next == NULL) return list; else { /* at least two elements on list */ SecondList = split(list); /* Note that as a side effect, half the elements are removed from list */ return merge(MergeSort(list), MergeSort(SecondList)); }
(3)
(4) }
Fig. 2.29. The merge sort algorithm.
✦
Example 2.25. Let us use merge sort on the list of single-digit numbers 742897721 We again omit commas between digits for succinctness. First, the list is split into two, by the call to split at line (3) of MergeSort. One of the resulting lists consists of the odd positions, and the other the evens; that is, list = 72971 and SecondList = 4872. At line (4), these lists are sorted, resulting in lists 12779 and 2478, and then merged to produce the sorted list 122477789. However, the sorting of the two half-sized lists does not occur by magic, but rather by the methodical application of the recursive algorithm. Initially, MergeSort splits the list on which it is called, if the list has length greater than 1. Figure 2.30(a) shows the recursive splitting of the lists until each list is of length 1. Then the split lists are merged, in pairs, going up the tree, until the entire list is sorted. This process is suggested in Fig. 2.30(b). However, it is worth noting that the splits and merges occur in a mixed order; not all splits are followed by all merges. For example, the first half list, 72971, is completely split and merged before we begin on the second half list, 4872. ✦
The Complete Program Figure 2.31 contains the complete merge sort program. It is analogous to the program in Fig. 2.3 that was based on selection sort. The function MakeList on line (1) reads each integer from the input and puts it into a linked list by a simple recursive algorithm, which we shall describe in detail in the next section. Line (2) of the main program contains the call to MergeSort, which returns a sorted list to PrintList. The function PrintList marches down the sorted list, printing each element.
EXERCISES 2.8.1: Show the result of applying the function merge to the lists (1, 2, 3, 4, 5) and (2, 4, 6, 8, 10).
82
ITERATION, INDUCTION, AND RECURSION
742897721
72971
4872
791
71
7
27
9
2
47
7
4
82
7
8
2
1 (a) Splitting.
122477789
12779
2478
179
17
7
27
9
2
47
7
4
28
7
8
2
1 (b) Merging. Fig. 2.30. Recursive splitting and merging.
2.8.2: Suppose we start with the list (8, 7, 6, 5, 4, 3, 2, 1). Show the sequence of calls to merge, split, and MergeSort that result. Multiway merge sort
2.8.3*: A multiway merge sort divides a list into k pieces of equal (or approximately equal) size, sorts them recursively, and then merges all k lists by comparing all their respective first elements and taking the smallest. The merge sort described in this section is for the case k = 2. Modify the program in Fig. 2.31 so that it becomes a multiway merge sort for the case k = 3. 2.8.4*: Rewrite the merge sort program to use the functions lt and key, described in Exercise 2.2.8, to compare elements of arbitrary type. 2.8.5: Relate each of the functions (a) merge (b) split (c) MakeList to Fig. 2.21. What is the appropriate notion of size for each of these functions?
SEC. 2.8
MERGE SORT: A RECURSIVE SORTING ALGORITHM
#include <stdio.h> #include <stdlib.h> typedef struct CELL *LIST; struct CELL { int element; LIST next; }; LIST LIST LIST LIST void
merge(LIST list1, LIST list2); split(LIST list); MergeSort(LIST list); MakeList(); PrintList(LIST list);
main() { LIST list; (1) (2)
list = MakeList(); PrintList(MergeSort(list)); } LIST MakeList() { int x; LIST pNewCell;
(3)
if (scanf("%d", &x) == EOF) return NULL; else { pNewCell = (LIST) malloc(sizeof(struct CELL)); pNewCell->next = MakeList(); pNewCell->element = x; return pNewCell; }
(4) (5) (6) (7) }
(8) (9) (10)
void PrintList(LIST list) { while (list != NULL) { printf("%d\n", list->element); list = list->next; } }
Fig. 2.31(a). A sorting program using merge sort (start).
83
84
ITERATION, INDUCTION, AND RECURSION
LIST MergeSort(LIST list) { LIST SecondList; if (list == NULL) return NULL; else if (list->next == NULL) return list; else { SecondList = split(list); return merge(MergeSort(list), MergeSort(SecondList)); } } LIST merge(LIST list1, LIST list2) { if (list1 == NULL) return list2; else if (list2 == NULL) return list1; else if (list1->element <= list2->element) { list1->next = merge(list1->next, list2); return list1; } else { list2->next = merge(list1, list2->next); return list2; } } LIST split(LIST list) { LIST pSecondCell; if (list == NULL) return NULL; else if (list->next == NULL) return NULL; else { pSecondCell = list->next; list->next = pSecondCell->next; pSecondCell->next = split(pSecondCell->next); return pSecondCell; } } Fig. 2.31(b). A sorting program using merge sort (conclusion).
✦ ✦ ✦ ✦
2.9
Proving Properties of Recursive Programs If we want to prove a certain property of a recursive function, we generally need to prove a statement about the effect of one call to that function. For example, that effect might be a relationship between the arguments and the return value, such as “the function, called with argument i, returns i!.” Frequently, we define a notion of the “size” of the arguments of a function and perform a proof by induction on this
SEC. 2.9
PROVING PROPERTIES OF RECURSIVE PROGRAMS
85
size. Some of the many possible ways in which size of arguments could be defined are
Size of arguments
1.
The value of some argument. For instance, for the recursive factorial program of Fig. 2.19, the appropriate size is the value of the argument n.
2.
The length of a list pointed to by some argument. The recursive function split of Fig. 2.27 is an example where the length of the list is the appropriate size.
3.
Some function of the arguments. For instance, we mentioned that the recursive selection sort of Fig. 2.22 performs an induction on the number of elements in the array that remain to be sorted. In terms of the arguments n and i, this function is n − i + 1. As another example, the appropriate size for the merge function of Fig. 2.24 is the sum of the lengths of the lists pointed to by the two arguments of the function.
Whatever notion of size we pick, it is essential that when a function is called with arguments of size s, it makes only function calls with arguments of size s − 1 or less. That requirement is so we can do an induction on the size to prove a property of the program. Further, when the size falls to some fixed value — say 0 — the function must make no recursive calls. This condition is so we can start off our inductive proof with a basis case.
✦
Example 2.26. Consider the factorial program of Fig. 2.19 in Section 2.7. The statement to prove by induction on i, for i ≥ 1, is
STATEMENT
S(i): When called with the value i for the argument n, fact re-
turns i!. BASIS. For i = 1, the test at line (1) of Fig. 2.19 causes the basis, line (2), to be executed. That results in the return value 1, which is 1!. INDUCTION. Assume S(i) to be true, that is, when called with some argument i ≥ 1, fact returns i!. Now, consider what happens when fact is called with i + 1 as the value of variable n. If i ≥ 1, then i + 1 is at least 2, so the inductive case, line (3), applies. The return value is thus n × f act(n − 1); or, since the variable n has the value i + 1, the result (i + 1) × f act(i) is returned. By the inductive hypothesis, fact(i) returns i!. Since (i + 1) × i! = (i + 1)!, we have proved the inductive step, that fact, with argument i + 1, returns (i + 1)!. ✦
✦
Example 2.27. Now, let us examine the function MakeList, one of the auxiliary routines in Fig. 2.31(a), in Section 2.8. This function creates a linked list to hold the input elements and returns a pointer to this list. We shall prove the following statement by induction on n ≥ 0, the number of elements in the input sequence.
86
ITERATION, INDUCTION, AND RECURSION
S(n): If x1 , x2 , ..., xn is the sequence of input elements, Makelist creates a linked list that contains x1 , x2 , ..., xn and returns a pointer to this list.
STATEMENT
BASIS. The basis is n = 0, that is, when the input sequence is empty. The test for EOF in line (3) of MakeList causes the return value to be set to NULL. Thus, MakeList correctly returns an empty list.
Suppose that S(n) is true for n ≥ 0, and consider what happens when MakeList is called on an input sequence of n + 1 elements. Suppose we have just read the first element x1 .
INDUCTION.
Line (4) of MakeList creates a pointer to a new cell c. Line (5) recursively calls Makelist to create, by the inductive hypothesis, a pointer to a linked list for the remaining n elements, x2 , x3 , . . . , xn . This pointer is put into the next field of c at line (5). Line (6) puts x1 into the element field of c. Line (7) returns the pointer created by line (4). This pointer points to a linked list for the n + 1 input elements, x1 , x2 , . . . , xn . We have proved the inductive step and conclude that MakeList works correctly on all inputs. ✦
✦
Example 2.28. For our last example, let us prove the correctness of the mergesort program of Fig. 2.29, assuming that the functions split and merge perform their respective tasks correctly. The induction will be on the length of the list that MergeSort is given as an argument. The statement to be proved by complete induction on n ≥ 0 is
S(n): If list is a list of length n when MergeSort is called, then MergeSort returns a sorted list of the same elements.
STATEMENT
We take the basis to be both S(0) and S(1). When list is of length 0, its value is NULL, and so the test of line (1) in Fig. 2.29 succeeds and the entire function returns NULL. Likewise, if list is of length 1, the test of line (2) succeeds, and the function returns list. Thus, MergeSort returns list when n is 0 or 1. This observation proves statements S(0) and S(1), because a list of length 0 or 1 is already sorted.
BASIS.
Suppose n ≥ 1 and S(i) is true for all i = 0, 1, . . . , n. We must prove S(n + 1). Thus, consider a list of length n + 1. Since n ≥ 1, this list is of length at least 2, so we reach line (3) in Fig. 2.29. There, split divides the list into two lists of length (n + 1)/2 if n + 1 is even, and of lengths (n/2) + 1 and n/2 if n + 1 is odd. Since n ≥ 1, none of these lists can be as long as n + 1. Thus, the inductive hypothesis applies to them, and we can conclude that the half-sized lists are correctly sorted by the recursive calls to MergeSort at line (4). Finally, the two sorted lists are merged into one list, which becomes the return value. We have assumed that merge works correctly, and so the resulting returned list is sorted. ✦ INDUCTION.
SEC. 2.10
SUMMARY OF CHAPTER 2
87
DefCell(int, CELL, LIST); int sum(LIST L) { if (L == NULL) sum = 0; else sum = L->element + sum(L->next); } int find0(LIST L) { if (L == NULL) find0 = FALSE; else if (L->element == 0) find0 = TRUE; else find0 = find0(L->next); } Fig. 2.32. Two recursive functions, sum and find0.
EXERCISES 2.9.1: Prove that the function PrintList in Fig. 2.31(b) prints the elements on the list that it is passed as an argument. What statement S(i) do you prove inductively? What is the basis value for i? 2.9.2: The function sum in Fig. 2.32 computes the sum of the elements on its given list (whose cells are of the usual type as defined by the macro DefCell of Section 1.6 and used in the merge-sort program of Section 2.8) by adding the first element to the sum of the remaining elements; the latter sum is computed by a recursive call on the remainder of the list. Prove that sum correctly computes the sum of the list elements. What statement S(i) do you prove inductively? What is the basis value for i? 2.9.3: The function find0 in Fig. 2.32 returns TRUE if at least one of the elements on its list is 0, and returns FALSE otherwise. It returns FALSE if the list is empty, returns TRUE if the first element is 0 and otherwise, makes a recursive call on the remainder of the list, and returns whatever answer is produced for the remainder. Prove that find0 correctly determines whether 0 is present on the list. What statement S(i) do you prove inductively? What is the basis value for i? 2.9.4*: Prove that the functions (a) merge of Fig. 2.24 and (b) split of Fig. 2.27 perform as claimed in Section 2.8. 2.9.5: Give an intuitive “least counterexample” proof of why induction starting from a basis including both 0 and 1 is valid. 2.9.6**: Prove the correctness of (your C implementation of) the recursive GCD algorithm of Exercise 2.7.8. ✦ ✦ ✦ ✦
2.10
Summary of Chapter 2 Here are the important ideas we should take from Chapter 2.
✦ ✦ ✦ ✦
2.11
88
ITERATION, INDUCTION, AND RECURSION
✦
Inductive proofs, recursive definitions, and recursive programs are closely related ideas. Each depends on a basis and an inductive step to “work.”
✦
In “ordinary” or “weak” inductions, successive steps depend only on the previous step. We frequently need to perform a proof by complete induction, in which each step depends on all the previous steps.
✦
There are several different ways to sort. Selection sort is a simple but slow sorting algorithm, and merge sort is a faster but more complex algorithm.
✦
Induction is essential to prove that a program or program fragment works correctly.
✦
Divide-and-conquer is a useful technique for designing some good algorithms, such as merge sort. It works by dividing the problem into independent subparts and then combining the results.
✦
Expressions are defined in a natural, recursive way in terms of their operands and operators. Operators can be classified by the number of arguments they take: unary (one argument), binary (two arguments), and k-ary (k arguments). Also, a binary operator appearing between its operands is infix, an operator appearing before its operands is prefix, and one appearing after its operands is postfix.
Bibliographic Notes for Chapter 2 An excellent treatment of recursion is Roberts [1986]. For more on sorting algorithms, the standard source is Knuth [1973]. Berlekamp [1968] tells about techniques — of which the error detection scheme in Section 2.3 is the simplest — for detecting and correcting errors in streams of bits. Berlekamp, E. R. [1968]. Algebraic Coding Theory, McGraw-Hill, New York. Knuth, D. E. [1973]. The Art of Computer Programming, Vol. III: Sorting and Searching, Addison-Wesley, Reading, Mass. Roberts, E. [1986]. Thinking Recursively, Wiley, New York.
3
CHAPTER
✦ ✦ ✦ ✦
The Running Time of Programs In Chapter 2, we saw two radically different algorithms for sorting: selection sort and merge sort. There are, in fact, scores of algorithms for sorting. This situation is typical: every problem that can be solved at all can be solved by more than one algorithm. How, then, should we choose an algorithm to solve a given problem? As a general rule, we should always pick an algorithm that is easy to understand, implement, and document. When performance is important, as it often is, we also need to choose an algorithm that runs quickly and uses the available computing resources efficiently. We are thus led to consider the often subtle matter of how we can measure the running time of a program or an algorithm, and what steps we can take to make a program run faster.
✦ ✦ ✦ ✦
3.1
What This Chapter Is About In this chapter we shall cover the following topics: ✦
The important performance measures for programs
✦
Methods for evaluating program performance
✦
“Big-oh” notation
✦
Estimating the running time of programs using the big-oh notation
✦
Using recurrence relations to evaluate the running time of recursive programs
The big-oh notation introduced in Sections 3.4 and 3.5 simplifies the process of estimating the running time of programs by allowing us to avoid dealing with constants that are almost impossible to determine, such as the number of machine instructions that will be generated by a typical C compiler for a given source program. We introduce the techniques needed to estimate the running time of programs in stages. In Sections 3.6 and 3.7 we present the methods used to analyze programs 89
90
THE RUNNING TIME OF PROGRAMS
with no function calls. Section 3.8 extends our capability to programs with calls to nonrecursive functions. Then in Sections 3.9 and 3.10 we show how to deal with recursive functions. Finally, Section 3.11 discusses solutions to recurrence relations, which are inductive definitions of functions that arise when we analyze the running time of recursive functions. ✦ ✦ ✦ ✦
Simplicity
Clarity
Efficiency
3.2
Choosing an Algorithm If you need to write a program that will be used once on small amounts of data and then discarded, then you should select the easiest-to-implement algorithm you know, get the program written and debugged, and move on to something else. However, when you need to write a program that is to be used and maintained by many people over a long period of time, other issues arise. One is the understandability, or simplicity, of the underlying algorithm. Simple algorithms are desirable for several reasons. Perhaps most important, a simple algorithm is easier to implement correctly than a complex one. The resulting program is also less likely to have subtle bugs that get exposed when the program encounters an unexpected input after it has been in use for a substantial period of time. Programs should be written clearly and documented carefully so that they can be maintained by others. If an algorithm is simple and understandable, it is easier to describe. With good documentation, modifications to the original program can readily be done by someone other than the original writer (who frequently will not be available to do them), or even by the original writer if the program was done some time earlier. There are numerous stories of programmers who wrote efficient and clever algorithms, then left the company, only to have their algorithms ripped out and replaced by something slower but more understandable by subsequent maintainers of the code. When a program is to be run repeatedly, its efficiency and that of its underlying algorithm become important. Generally, we associate efficiency with the time it takes a program to run, although there are other resources that a program sometimes must conserve, such as 1.
The amount of storage space taken by its variables.
2.
The amount of traffic it generates on a network of computers.
3.
The amount of data that must be moved to and from disks.
For large problems, however, it is the running time that determines whether a given program can be used, and running time is the main topic of this chapter. We shall, in fact, take the efficiency of a program to mean the amount of time it takes, measured as a function of the size of its input. Often, understandability and efficiency are conflicting aims. For example, the reader who compares the selection sort program of Fig. 2.3 with the merge sort program of Fig. 2.32 will surely agree that the latter is not only longer, but quite a bit harder to understand. That would still be true even if we summarized the explanation given in Sections 2.2 and 2.8 by placing well-thought-out comments in the programs. As we shall learn, however, merge sort is much more efficient than selection sort, as long as the number of elements to be sorted is a hundred or more. Unfortunately, this situation is quite typical: algorithms that are efficient for large
SEC. 3.3
MEASURING RUNNING TIME
91
amounts of data tend to be more complex to write and understand than are the relatively inefficient algorithms. The understandability, or simplicity, of an algorithm is somewhat subjective. We can overcome lack of simplicity in an algorithm, to a certain extent, by explaining the algorithm well in comments and program documentation. The documentor should always consider the person who reads the code and its comments: Is a reasonably intelligent person likely to understand what is being said, or are further explanation, details, definitions, and examples needed? On the other hand, program efficiency is an objective matter: a program takes what time it takes, and there is no room for dispute. Unfortunately, we cannot run the program on all possible inputs — which are typically infinite in number. Thus, we are forced to make measures of the running time of a program that summarize the program’s performance on all inputs, usually as a single expression such as “n2 .” How we can do so is the subject matter of the balance of this chapter.
✦ ✦ ✦ ✦
3.3
Measuring Running Time Once we have agreed that we can evaluate a program by measuring its running time, we face the problem of determining what the running time actually is. The two principal approaches to summarizing the running time are 1.
Benchmarking
2.
Analysis
We shall consider each in turn, but the primary emphasis of this chapter is on the techniques for analyzing a program or an algorithm.
Benchmarking When comparing two or more programs designed to do the same set of tasks, it is customary to develop a small collection of typical inputs that can serve as benchmarks. That is, we agree to accept the benchmark inputs as representative of the job mix; a program that performs well on the benchmark inputs is assumed to perform well on all inputs. For example, a benchmark to evaluate sorting programs might contain one small set of numbers, such as the first 20 digits of π; one medium set, such as the set of zip codes in Texas; and one large set, such as the set of phone numbers in the Brooklyn telephone directory. We might also want to check that a program works efficiently (and correctly) when given an empty set of elements to sort, a singleton set, and a list that is already in sorted order. Interestingly, some sorting algorithms perform poorly when given a list of elements that is already sorted.1
1
Neither selection sort nor merge sort is among these; they take approximately the same time on a sorted list as they would on any other list of the same length.
92
THE RUNNING TIME OF PROGRAMS
The 90-10 Rule
Profiling
Locality and hot spots
In conjunction with benchmarking, it is often useful to determine where the program under consideration is spending its time. This method of evaluating program performance is called profiling and most programming environments have tools called profilers that associate with each statement of a program a number that represents the fraction of the total time taken executing that particular statement. A related utility, called a statement counter, is a tool that determines for each statement of a source program the number of times that statement is executed on a given set of inputs. Many programs exhibit the property that most of their running time is spent in a small fraction of the source code. There is an informal rule that states 90% of the running time is spent in 10% of the code. While the exact percentage varies from program to program, the “90-10 rule” says that most programs exhibit significant locality in where the running time is spent. One of the easiest ways to speed up a program is to profile it and then apply code improvements to its “hot spots,” which are the portions of the program in which most of the time is spent. For example, we mentioned in Chapter 2 that one might speed up a program by replacing a recursive function with an equivalent iterative one. However, it makes sense to do so only if the recursive function occurs in those parts of the program where most of the time is being spent. As an extreme case, even if we reduce to zero the time taken by the 90% of the code in which only 10% of the time is spent, we will have reduced the overall running time of the program by only 10%. On the other hand, cutting in half the running time of the 10% of the program where 90% of the time is spent reduces the overall running time by 45%.
Analysis of a Program To analyze a program, we begin by grouping inputs according to size. What we choose to call the size of an input can vary from program to program, as we discussed in Section 2.9 in connection with proving properties of recursive programs. For a sorting program, a good measure of the size is the number of elements to be sorted. For a program that solves n linear equations in n unknowns, it is normal to take n to be the size of the problem. Other programs might use the value of some particular input, or the length of a list that is an input to the program, or the size of an array that is an input, or some combination of quantities such as these.
Running Time
Linear-time algorithm
It is convenient to use a function T (n) to represent the number of units of time taken by a program or an algorithm on any input of size n. We shall call T (n) the running time of the program. For example, a program may have a running time T (n) = cn, where c is some constant. Put another way, the running time of this program is linearly proportional to the size of the input on which it is run. Such a program or algorithm is said to be linear time, or just linear. We can think of the running time T (n) as the number of C statements executed by the program or as the length of time taken to run the program on some standard computer. Most of the time we shall leave the units of T (n) unspecified. In fact,
SEC. 3.3
MEASURING RUNNING TIME
93
as we shall see in the next section, it makes sense to talk of the running time of a program only as some (unknown) constant factor times T (n). Quite often, the running time of a program depends on a particular input, not just on the size of the input. In these cases, we define T (n) to be the worst-case running time, that is, the maximum running time on any input among all inputs of size n. Another common performance measure is Tavg (n), the average running time of the program over all inputs of size n. The average running time is sometimes a more realistic measure of what performance one will see in practice, but it is often much harder to compute than the worst-case running time. The notion of an “average” running time also implies that all inputs of size n are equally likely, which may or may not be true in a given situation.
Worst and average-case running time
✦
Example 3.1. Let us estimate the running time of the SelectionSort fragment shown in Fig. 3.1. The statements have the original line numbers from Fig. 2.2. The purpose of the code is to set small to the index of the smallest of the elements found in the portion of the array A from A[i] through A[n-1]. (2) (3) (4) (5)
small = i; for(j = i+1; j < n; j++) if (A[j] < A[small]) small = j; Fig. 3.1. Inner loop of selection sort.
To begin, we need to develop a simple notion of time units. We shall examine the issue in detail later, but for the moment, the following simple scheme is sufficient. We shall count one time unit each time we execute an assignment statement. At line (3), we count one unit for initializing j at the beginning of the for-loop, one unit for testing whether j < n, and one unit for incrementing j, each time we go around the loop. Finally, we charge one unit each time we perform the test of line (4). First, let us consider the body of the inner loop, lines (4) and (5). The test of line (4) is always executed, but the assignment at line (5) is executed only if the test succeeds. Thus, the body takes either 1 or 2 time units, depending on the data in array A. If we want to take the worst case, then we can assume that the body takes 2 units. We go around the for-loop n − i − 1 times, and each time around we execute the body (2 units), then increment j and test whether j < n (another 2 units). Thus, the number of time units spent going around the loop is 4(n − i − 1). To this number, we must add 1 for initializing small at line (2), 1 for initializing j at line (3), and 1 for the first test j < n at line (3), which is not associated with the end of any iteration of the loop. Hence, the total time taken by the program fragment in Fig. 3.1 is 4(n − i) − 1. It is natural to regard the “size” m of the data on which Fig. 3.1 operates as m = n − i, since that is the length of the array A[i..n-1] on which it operates. Then the running time, which is 4(n − i) − 1, equals 4m − 1. Thus, the running time T (m) for Fig. 3.1 is 4m − 1. ✦
94
THE RUNNING TIME OF PROGRAMS
Comparing Different Running Times Suppose that for some problem we have the choice of using a linear-time program A whose running time is TA (n) = 100n and a quadratic-time program B whose running time is TB (n) = 2n2 . Let us suppose that both these running times are the number of milliseconds taken on a particular computer on an input of size n.2 The graphs of the running times are shown in Fig. 3.2. 20,000 TB = 2n2
15,000
T (n)
10,000 TA = 100n
5,000
0 0
20
40
60
80
100
Input size n Fig. 3.2. Running times of a linear and a quadratic program.
From Fig. 3.2 we see that for inputs of size less than 50, program B is faster than program A. When the input becomes larger than 50, program A becomes faster, and from that point on, the larger the input, the bigger the advantage A has over B. For inputs of size 100, A is twice as fast as B, and for inputs of size 1000, A is 20 times as fast. The functional form of a program’s running time ultimately determines how big a problem we can solve with that program. As the speed of computers increases, we get bigger improvements in the sizes of problems that we can solve with programs whose running times grow slowly than with programs whose running times rise rapidly. Again, assuming that the running times of the programs shown in Fig. 3.2 are in milliseconds, the table in Fig. 3.3 indicates how large a problem we can solve with each program on the same computer in various amounts of time given in seconds. For example, suppose we can afford 100 seconds of computer time. If computers become 10 times as fast, then in 100 seconds we can handle problems of the size that used to require 1000 seconds. With algorithm A, we can now solve problems 10 times as large, but with algorithm B we can only solve problems about 3 times as large. Thus, as computers continue to get faster, we gain an even more significant advantage by using algorithms and programs with lower growth rates. 2
This situation is not too dissimilar to the situation where algorithm A is merge sort and algorithm B is selection sort. However, the running time of merge sort grows as n log n, as we shall see in Section 3.10.
SEC. 3.3
MEASURING RUNNING TIME
95
Never Mind Algorithm Efficiency; Just Wait a Few Years Frequently, one hears the argument that there is no need to improve the running time of algorithms or to select efficient algorithms, because computer speeds are doubling every few years and it will not be long before any algorithm, however inefficient, will take so little time that one will not care. People have made this claim for many decades, yet there is no limit in sight to the demand for computational resources. Thus, we generally reject the view that hardware improvements will make the study of efficient algorithms superfluous. There are situations, however, when we need not be overly concerned with efficiency. For example, a school may, at the end of each term, transcribe grades reported on electronically readable grade sheets to student transcripts, all of which are stored in a computer. The time this operation takes is probably linear in the number of grades reported, like the hypothetical algorithm A. If the school replaces its computer by one 10 times as fast, it can do the job in one-tenth the time. It is very unlikely, however, that the school will therefore enroll 10 times as many students, or require each student to take 10 times as many classes. The computer speedup will not affect the size of the input to the transcript program, because that size is limited by other factors. On the other hand, there are some problems that we are beginning to find approachable with emerging computing resources, but whose “size” is too great to handle with existing technology. Some of these problems are natural language understanding, computer vision (understanding of digitized pictures), and “intelligent” interaction between computers and humans in all sorts of endeavors. Speedups, either through improved algorithms or from machine improvements, will enhance our ability to deal with these problems in the coming years. Moreover, when they become “simple” problems, a new generation of challenges, which we can now only barely imagine, will take their place on the frontier of what it is possible to do with computers.
TIME sec.
MAXIMUM PROBLEM SIZE SOLVABLE WITH PROGRAM A
MAXIMUM PROBLEM SIZE SOLVABLE WITH PROGRAM B
1 10 100 1000
10 100 1000 10000
22 70 223 707
Fig. 3.3. Problem size as a function of available time.
EXERCISES 3.3.1: Consider the factorial program fragment in Fig. 2.13, and let the input size be the value of n that is read. Counting one time unit for each assignment, read, and write statement, and one unit each time the condition of the while-statement is tested, compute the running time of the program.
96
THE RUNNING TIME OF PROGRAMS
3.3.2: For the program fragments of (a) Exercise 2.5.1 and (b) Fig. 2.14, give an appropriate size for the input. Using the counting rules of Exercise 3.3.1, determine the running times of the programs. 3.3.3: Suppose program A takes 2n /1000 units of time and program B takes 1000n2 units. For what values of n does program A take less time than program B? 3.3.4: For each of the programs of Exercise 3.3.3, how large a problem can be solved in (a) 106 time units, (b) 109 time units, and (c) 1012 time units? 3.3.5: Repeat Exercises 3.3.3 and 3.3.4 if program A takes 1000n4 time units and program B takes n10 time units. ✦ ✦ ✦ ✦
3.4
Big-Oh and Approximate Running Time Suppose we have written a C program and have selected the particular input on which we would like it to run. The running time of the program on this input still depends on two factors: 1.
The computer on which the program is run. Some computers execute instructions more rapidly than others; the ratio between the speeds of the fastest supercomputers and the slowest personal computers is well over 1000 to 1.
2.
The particular C compiler used to generate a program for the computer to execute. Different programs can take different amounts of time to execute on the same machine, even though the programs have the same effect.
As a result, we cannot look at a C program and its input and say, “This task will take 3.21 seconds,” unless we know which machine and which compiler will be used. Moreover, even if we know the program, the input, the machine, and the compiler, it is usually far too complex a task to predict exactly the number of machine instructions that will be executed. For these reasons, we usually express the running time of a program using “big-oh” notation, which is designed to let us hide constant factors such as 1.
The average number of machine instructions a particular compiler generates.
2.
The average number of machine instructions a particular machine executes per second.
For example, instead of saying, as we did in Example 3.1, that the SelectionSort fragment we studied takes time 4m − 1 on an array of length m, we would say that it takes O(m) time, which is read “big-oh of m” or just “oh of m,” and which informally means “some constant times m.” The notion of “some constant times m” not only allows us to ignore unknown constants associated with the compiler and the machine, but also allows us to make some simplifying assumptions. In Example 3.1, for instance, we assumed that all assignment statements take the same amount of time, and that this amount of time was also taken by the test for termination in the for-loop, the incrementation of j around the loop, the initialization, and so on. Since none of these assumptions is valid in practice, the constants 4 and −1 in the running-time formula T (m) = 4m−1 are at best approximations to the truth. It would be more appropriate to describe T (m) as “some constant times m, plus or minus another constant” or even as “at
SEC. 3.4
BIG-OH AND APPROXIMATE RUNNING TIME
97
most proportional to m.” The notation O(m) enables us to make these statements without getting involved in unknowable or meaningless constants. On the other hand, representing the running time of the fragment as O(m) does tell us something very important. It says that the time to execute the fragment on progressively larger arrays grows linearly, like the hypothetical Program A of Figs. 3.2 and 3.3 discussed at the end of Section 3.3. Thus, the algorithm embodied by this fragment will be superior to competing algorithms whose running time grows faster, such as the hypothetical Program B of that discussion.
Definition of Big-Oh We shall now give a formal definition of the notion of one function being “big-oh” of another. Let T (n) be a function, which typically is the running time of some program, measured as a function of the input size n. As befits a function that measures the running time of a program, we shall assume that 1.
The argument n is restricted to be a nonnegative integer, and
2.
The value T (n) is nonnegative for all arguments n.
Let f (n) be some function defined on the nonnegative integers n. We say that “T (n) is O f (n) ” if T (n) is at most a constant times f (n), except possibly for some small values of n. Formally, we say that T (n) is O f (n) if there exists an integer n0 and a constant c > 0 such that for all integers n ≥ n0 , we have T (n) ≤ cf (n). We call the pair n0 and c witnesses to the fact that T (n) is O f (n) . The witnesses “testify” to the big-oh relationship of T (n) and f (n) in a form of proof that we shall next demonstrate.
Witnesses
Proving Big-Oh Relationships We can apply the definition of “big-oh” to prove that T (n) is O f (n) for particular functions T and f . We do so by exhibiting a particular choice of witnesses n0 and c and then proving that T (n) ≤ cf (n). The proof must assume only that n is a nonnegative integer and that n is at least as large as our chosen n0 . Usually, the proof involves algebra and manipulation of inequalities.
✦ Quadratic running time
Example 3.2. Suppose we have a program whose running time is T (0) = 1, T (1) = 4, T (2) = 9, and in general T (n) = (n + 1)2 . We can say that T (n) is O(n2 ), or that T (n) is quadratic, because we can choose witnesses n0 = 1 and c = 4. We then need to prove that (n + 1)2 ≤ 4n2 , provided n ≥ 1. In proof, expand (n + 1)2 as n2 + 2n + 1. As long as n ≥ 1, we know that n ≤ n2 and 1 ≤ n2 . Thus n2 + 2n + 1 ≤ n2 + 2n2 + n2 = 4n2
98
THE RUNNING TIME OF PROGRAMS
Template for Big-Oh Proofs Remember: all big-oh proofs follow essentially the same form. Only the algebraic manipulation varies. We need to do only two things to have a proof that T (n) is O f (n) . 1.
State the witnesses n0 and c. These witnesses must be specific constants, e.g., n0 = 47 and c = 12.5. Also, n0 must be a nonnegative integer, and c must be a positive real number.
2.
By appropriate algebraic manipulation, show that if n ≥ n0 then T (n) ≤ cf (n), for the particular witnesses n0 and c chosen.
Alternatively, we could pick witnesses n0 = 3 and c = 2, because, as the reader may check, (n + 1)2 ≤ 2n2 , for all n ≥ 3. However, we cannot pick n0 = 0 with any c, because with n = 0, we would have to show that (0 + 1)2 ≤ c02 , that is, that 1 is less than or equal to c times 0. Since c × 0 = 0 no matter what c we pick, and 1 ≤ 0 is false, we are doomed if we pick n0 = 0. That doesn’t matter, however, because in order to show that (n + 1)2 is O(n2 ), we had only to find one choice of witnesses n0 and c that works. ✦ It may seem odd that although (n + 1)2 is larger than n2 , we can still say that (n + 1)2 is O(n2 ). In fact, we can also say that (n + 1)2 is big-oh of any fraction of n2 , for example, O(n2 /100). To see why, choose witnesses n0 = 1 and c = 400. Then if n ≥ 1, we know that (n + 1)2 ≤ 400(n2 /100) = 4n2
by the same reasoning as was used in Example 3.2. The general principles underlying these observations are that 1.
Constant factors don’t matter. For any positive constant d and any function T (n), T (n) is O dT (n) , regardless of whether d is a large number or a very small fraction, as long as d > 0. To see why, choose witnesses n0 = 0 and c = 1/d.3 Then if we know that T (n) ≤ c dT (n) , since cd = 1. Likewise, T (n) is O f (n) , then we also know that T (n) is O df (n) for any d > 0, even a very small d. The reason is that we know that T (n) ≤ c1 f (n) for some constant c1 and all n ≥ n0 . If we choose c = c1 /d, we can see that T (n) ≤ c df (n) for n ≥ n0 .
2.
Low-order terms don’t matter. Suppose T (n) is a polynomial of the form ak nk + ak−1 nk−1 + · · · + a2 n2 + a1 n + a0 where the leading coefficient, ak , is positive. Then we can throw away all terms but the first (the term with the highest exponent, k) and, by rule (1), ignore the constant ak , replacing it by 1. That is, we can conclude T (n) is O(nk ). In proof, let n0 = 1, and let c be the sum of all the positive coefficients among the ai ’s, 0 ≤ i ≤ k. If a coefficient aj is 0 or negative, then surely aj nj ≤ 0. If 3
Note that although we are required to choose constants as witnesses, not functions, there is nothing wrong with choosing c = 1/d, because d itself is some constant.
SEC. 3.4
BIG-OH AND APPROXIMATE RUNNING TIME
99
Fallacious Arguments About Big-Oh The definition of “big-oh” is tricky, in that it requires us, after examining T (n) and f (n), to pick witnesses n0 and c once and for all, and then to show that T (n) ≤ cf (n) for all n ≥ n0 . It is not permitted to pick c and/or n0 anew for each value of n. For example, one occasionally sees the following fallacious “proof” that n2 is O(n). “Pick n0 = 0, and for each n, pick c = n. Then n2 ≤ cn.” This argument is invalid, because we are required to pick c once and for all, without knowing n.
aj is positive, then aj nj ≤ aj nk , for all j < k, as long as n ≥ 1. Thus, T (n) is no greater than nk times the sum of the positive coefficients, or cnk .
✦
Example 3.3. As an example of rule (1) (“constants don’t matter”), we can see that 2n3 is O(.001n3 ). Let n0 = 0 and c = 2/.001 = 2000. Then clearly 2n3 ≤ 2000(.001n3) = 2n3 , for all n ≥ 0. As an example of rule (2) (“low order terms don’t matter”), consider the polynomial T (n) = 3n5 + 10n4 − 4n3 + n + 1
The highest-order term is n5 , and we claim that T (n) is O(n5 ). To check the claim, let n0 = 1 and let c be the sum of the positive coefficients. The terms with positive coefficients are those with exponents 5, 4, 1, and 0, whose coefficients are, respectively, 3, 10, 1, and 1. Thus, we let c = 15. We claim that for n ≥ 1, 3n5 + 10n4 − 4n3 + n + 1 ≤ 3n5 + 10n5 + n5 + n5 = 15n5
Growth rate
(3.1)
We can check that inequality (3.1) holds by matching the positive terms; that is, 3n5 ≤ 3n5 , 10n4 ≤ 10n5 , n ≤ n5 , and 1 ≤ n5 . Also, the negative term on the left side of (3.1) can be ignored, since −4n3 ≤ 0. Thus, the left side of (3.1), which is T (n), is less than or equal to the right side, which is 15n5 , or cn5 . We can conclude that T (n) is O(n5 ). In fact, the principle that low-order terms can be deleted applies not only to polynomials, but to any sum of expressions. That is, if the ratio h(n)/g(n) approaches 0 as n approaches infinity, then h(n) “grows more slowly” than g(n), or “has a lower growth rate” than g(n), and we may neglect h(n). That is, g(n) + h(n) is O g(n) . For example, let T (n) = 2n + n3 . It is known that every polynomial, such as 3 n , grows more slowly than every exponential, such as 2n . Since n3 /2n approaches 0 as n increases, we can throw away the lower-order term and conclude that T (n) is O(2n ). To prove formally that 2n + n3 is O(2n ), let n0 = 10 and c = 2. We must show that for n ≥ 10, we have 2 n + n3 ≤ 2 × 2 n
If we subtract 2n from both sides, we see it is sufficient to show that for n ≥ 10, it is the case that n3 ≤ 2n . For n = 10 we have 210 = 1024 and 103 = 1000, and so n3 ≤ 2n for n = 10. Each time we add 1 to n, 2n doubles, while n3 is multiplied by a quantity (n+1)3 /n3
100
THE RUNNING TIME OF PROGRAMS
that is less than 2 when n ≥ 10. Thus, as n increases beyond 10, n3 becomes progressively less than 2n . We conclude that n3 ≤ 2n for n ≥ 10, and thus that 2n + n3 is O(2n ). ✦
Proofs That a Big-Oh Relationship Does Not Hold If a big-oh relationship holds between two functions, then we can prove it by finding witnesses. However, what if some function T (n) is not big-oh of some other function f (n)? Can we ever hope to be sure that there is not such a big-oh relationship? The answer is that quite frequently we can prove a particular function T (n) is not O f (n) . The method of proof is to assume that witnesses n0 and c exist, and derive a contradiction. The following is an example of such a proof.
✦
Example 3.4. In the box on “Fallacious Arguments About Big-Oh,” we claimed that n2 is not O(n). We can show this claim as follows. Suppose it were. Then there would be witnesses n0 and c such that n2 ≤ cn for all n ≥ n0 . But if we pick n1 equal to the larger of 2c and n0 , then the inequality (n1 )2 ≤ cn1
(3.2) 2
must hold (because n1 ≥ n0 and n ≤ cn allegedly holds for all n ≥ n0 ). If we divide both sides of (3.2) by n1 , we have n1 ≤ c. However, we also chose n1 to be at least 2c. Since witness c must be positive, n1 cannot be both less than c and greater than 2c. Thus, witnesses n0 and c that would show n2 to be O(n) do not exist, and we conclude that n2 is not O(n). ✦
EXERCISES 3.4.1: Consider the four functions f1 : f2 : f3 : f4 :
n2 n3 n2 if n is odd, and n3 if n is even n2 if n is prime, and n3 if n is composite
For each i and j equal to 1, 2, 3, 4, determine whether fi (n) is O fj (n) . Either give values n0 and c that prove the big-oh relationship, or assume that there are such values n0 and c, and then derive a contradiction to prove that fi (n) is not O fj (n) . Hint : Remember that all primes except 2 are odd. Also remember that there are an infinite number of primes and an infinite number of composite numbers (nonprimes). 3.4.2: Following are some big-oh relationships. For each, give witnesses n0 and c that can be used to prove the relationship. Choose your witnesses to be minimal, in the sense that n0 − 1 and c are not witnesses, and if d < c, then n0 and d are not witnesses. a) b) c) d) e)*
n2 is O(.001n3 ) 25n4 − 19n3 + 13n2 − 106n + 77 is O(n4 ) 2n+10 is O(2n ) n10 is O(3n )√ log2 n is O( n)
SEC. 3.5
SIMPLIFYING BIG-OH EXPRESSIONS
101
Template for Proofs That a Big-Oh Relationship Is False The following is an outline of typical proofs that a function T (n) is not O f (n) . Example 3.4 illustrates such a proof. 1.
Start by supposing that there were witnesses n0 and c such that for all n ≥ n0 , we have f (n) ≤ cg(n). Here, n0 and c are symbols standing for unknown witnesses.
2.
Define a particular integer n1 , expressed in terms of n0 and c (for example, n1 = max(n0 , 2c) was chosen in Example 3.4). This n1 will be the value of n for which we show T (n1 ) ≤ cf (n1 ) is false.
3.
Show that for the chosen n1 we have n1 ≥ n0 . This part can be very easy, since we may choose n1 to be at least n0 in step (2).
4.
Claim that because n1 ≥ n0 , we must have T (n1 ) ≤ cf (n1 ).
5.
Derive a contradiction by showing that for the chosen n1 we have T (n1 ) > cf (n1 ). Choosing n1 in terms of c can make this part easy, as it was in Example 3.4.
3.4.3*: Prove that if f (n) ≤ g(n) for all n, then f (n) + g(n) is O g(n) . 3.4.4**: Suppose that f (n) is O g(n) and g(n) is O f (n) . What can you say about f (n) and g(n)? Is it necessarily true that f (n) = g(n)? Does the limit f (n)/g(n) as n goes to infinity necessarily exist? ✦ ✦ ✦ ✦
3.5
Simplifying Big-Oh Expressions As we saw in the previous section, it is possible to simplify big-oh expressions by dropping constant factors and low-order terms. We shall see how important it is to make such simplifications when we analyze programs. It is common for the running time of a program to be attributable to many different statements or program fragments, but it is also normal for a few of these pieces to account for the bulk of the running time (by the “90-10” rule). By dropping low-order terms, and by combining equal or approximately equal terms, we can often greatly simplify the expressions for running time.
The Transitive Law for Big-Oh Expressions To begin, we shall take up a useful rule for thinking about big-oh expressions. A relationship like ≤ is said to be transitive, because it obeys the law “if A ≤ B and B ≤ C, then A ≤ C.” For example, since 3 ≤ 5 and 5 ≤ 10, we can be sure that 3 ≤ 10. The relationship “is big-oh of” is another example of a transitive relationship. That is, if f (n) is O g(n) and g(n) is O h(n) , it follows that f (n) is O h(n) . To see why, first suppose that f (n) is O g(n) . Then there are witnesses n1 and c1 such that f (n) ≤ c1 g(n) for all n ≥ n1 . Similarly, if g(n) is O h(n) , then there are witnesses n2 and c2 such that g(n) ≤ c2 h(n) for all n ≥ n2 .
102
THE RUNNING TIME OF PROGRAMS
Polynomial and Exponential Big-Oh Expressions The degree of a polynomial is the highest exponent found among its terms. For example, the degree of the polynomial T (n) mentioned in Examples 3.3 and 3.5 is 5, because 3n5 is its highest-order term. From the two principles we have enunciated (constant factors don’t matter, and low-order terms don’t matter), plus the transitive law for big-oh expressions, we know the following: 1. 2. 3.
Exponential
4.
If p(n) and q(n) are polynomials and the degree of q(n) is as high as or higher than the degree of p(n), then p(n) is O q(n) . If the degree of q(n) is lower than the degree of p(n), then p(n) is not O q(n) .
Exponentials are expressions of the form an for a > 1. Every exponential grows faster than every polynomial. That is, we can show for any polynomial p(n) that p(n) is O(an ). For example, n5 is O (1.01)n . Conversely, no exponential an , for a > 1, is O p(n) for any polynomial p(n).
Let n0 be the larger of n1 and n2 , and let c = c1 c2 . We claim that n0 and c are witnesses to the fact that f (n) is O h(n) . For suppose n ≥ n0 . Since n0 = max(n1 , n2 ), we know that n ≥ n1 and n ≥ n2 . Therefore, f (n) ≤ c1 g(n) and g(n) ≤ c2 h(n). Now substitute c2 h(n) for g(n) in the inequality f (n) ≤ c1 g(n), to prove f (n) ≤ c1 c2 h(n). This inequality shows f (n) is O h(n) . ✦
Example 3.5. We know from Example 3.3 that T (n) = 3n5 + 10n4 − 4n3 + n + 1
is O(n5 ). We also know from the rule that “constant factors don’t matter” that n5 is O(.01n5 ). By the transitive law for big-oh, we know that T (n) is O(.01n5 ). ✦
Describing the Running Time of a Program We defined the running time T (n) of a program to be the maximum number of time units taken on any input of size n. We also said that determining a precise formula for T (n) is a difficult, if not impossible, task. Frequently, we can simplify matters considerably by using a big-oh expression O(f (n)) as an upper bound on T (n). For example, an upper bound on the running time T (n) of SelectionSort is an2 , for some constant a and any n ≥ 1; we shall demonstrate this fact in Section 3.6. Then we can say the running time of SelectionSort is O(n2 ). That statement is intuitively the most useful one to make, because n2 is a very simple function, and stronger statements about other simple functions, like “T (n) is O(n),” are false. However, because of the nature of big-oh notation, we can also state that the running time T (n) is O(.01n2 ), or O(7n2 −4n+26), or in fact big-oh of any quadratic polynomial. The reason is that n2 is big-oh of any quadratic, and so the transitive law plus the fact that T (n) is O(n2 ) tells us T (n) is big-oh of any quadratic. Worse, n2 is big-oh of any polynomial of degree 3 or higher, or of any exponential. Thus, by transitivity again, T (n) is O(n3 ), O(2n + n4 ), and so on. However,
SEC. 3.5
SIMPLIFYING BIG-OH EXPRESSIONS
103
we shall explain why O(n2 ) is the preferred way to express the running time of SelectionSort.
Tightness First, we generally want the “tightest” big-oh upper bound we can prove. That is, if T (n) is O(n2 ), we want to say so, rather than make the technically true but weaker statement that T (n) is O(n3 ). On the other hand, this way lies madness, because if we like O(n2 ) as an expression of running time, we should like O(0.5n2 ) even better, because it is “tighter,” and we should like O(.01n2 ) still more, and so on. However, since constant factors don’t matter in big-oh expressions, there is really no point in trying to make the estimate of running time “tighter” by shrinking the constant factor. Thus, whenever possible, we try to use a big-oh expression that has a constant factor 1. Figure 3.4 lists some of the more common running times for programs and their informal names. Note in particular that O(1) is an idiomatic shorthand for “some constant,” and we shall use O(1) repeatedly for this purpose. BIG-OH
INFORMAL NAME
O(1) O(log n) O(n) O(n log n) O(n2 ) O(n3 ) O(2n )
constant logarithmic linear n log n quadratic cubic exponential
Fig. 3.4. Informal names for some common big-oh running times. Tight bound
1. 2.
✦
More precisely, we shall say that f (n) is a tight big-oh bound on T (n) if T (n) is O f (n) , and If T (n) is O g(n) , then it is also true that f (n) is O g(n) . Informally, we cannot find a function g(n) that grows at least as fast as T (n) but grows slower than f (n).
Example 3.6. Let T (n) = 2n2 + 3n and f (n) = n2 . We claim that f (n) is
a tight bound on T (n). To see why, suppose T (n) is O g(n) . Then there are constants c and n0 such that for all n ≥ n0 , we have T (n) = 2n2 + 3n ≤ cg(n). Then g(n) ≥ (2/c)n2 for n ≥n0 . Since f (n) is n2 , we have f (n) ≤ (c/2)g(n) for n ≥ n0 . Thus, f (n) is O g(n) . On the other hand, f (n) = n3 is not a tight big-oh bound on T (n). Now we can pick g(n) = n2 . We have seen that T (n) is O g(n) , but we cannot show that f (n) is O g(n) , since n3 is not O(n2 ). Thus, n3 is not a tight big-oh bound on T (n). ✦
104
THE RUNNING TIME OF PROGRAMS
Simplicity
Simple function
✦
The other goal in our choice of a big-oh bound is simplicity in the expression of the function. Unlike tightness, simplicity can sometimes be a matter of taste. However, we shall generally regard a function f (n) as simple if 1.
It is a single term and
2.
The coefficient of that term is 1.
Example 3.7. The function n2 is simple; 2n2 is not simple because the coefficient is not 1, and n2 + n is not simple because there are two terms. ✦ There are some situations, however, where the tightness of a big-oh upper bound and simplicity of the bound are conflicting goals. The following is an example where the simple bound doesn’t tell the whole story. Fortunately, such cases are rare in practice.
Tightness and simplicity may conflict
int PowersOfTwo(int n) { int i; (1) (2) (3) (4)
i = 0; while (n%2 == 0) { n = n/2; i++; } return i;
(5) }
Fig. 3.5. Counting factors of 2 in a positive integer n.
✦
Example 3.8. Consider the function PowersOfTwo in Fig. 3.5, which takes a positive argument n and counts the number times 2 divides n. That is, the test of line (2) asks whether n is even and, if so, removes a factor of 2 at line (3) of the loop body. Also in the loop, we increment i, which counts the number of factors we have removed from the original value of n. Let the size of the input be the value of n itself. The body of the while-loop consists of two C assignment statements, lines (3) and (4), and so we can say that the time to execute the body once is O(1), that is, some constant amount of time, independent of n. If the loop is executed m times, then the total time spent going around the loop will be O(m), or some amount of time that is proportional to m. To this quantity we must add O(1), or some constant, for the single executions of lines (1) and (5), plus the first test of the while-condition, which is technically not part of any loop iteration. Thus, the time spent by the program is O(m) + O(1). Following our rule that low-order terms can be neglected, the time is O(m), unless m = 0, in which case it is O(1). Put another way, the time spent on input n is proportional to 1 plus the number of times 2 divides n.
SEC. 3.5
SIMPLIFYING BIG-OH EXPRESSIONS
105
Using Big-Oh Notation in Mathematical Expressions Strictly speaking, the only mathematically correct way to use a big-oh expression is after the word “is,” as in “2n2 is O(n3 ).” However, in Example 3.8, and for the remainder of the chapter, we shall take a liberty and use big-oh expressions as operands of addition and other arithmetic operators, as in O(n) + O(n2 ). We should interpret a big-oh expression used this way as meaning “some function that is big-oh of.” For example, O(n) + O(n2 ) means “the sum of some linear function and some quadratic function.” Also, O(n) + T (n) should be interpreted as the sum of some linear function and the particular function T (n).
How many times does 2 divide n? For every odd n, the answer is 0, so the function PowersOfTwo takes time O(1) on every odd n. However, when n is a power of 2 — that is, when n = 2k for some k — 2 divides n exactly k times. When n = 2k , we may take logarithms to base 2 of both sides and conclude that log2 n = k. That is, m is at most logarithmic in n, or m = O(log n).4 4 f (n)
3 2 1
1
2
3
4
5
6
7
8
9
10
Fig. 3.6. The function f (n) = m(n) + 1, where m(n) is the number of times 2 divides n.
We may thus say that the running time of PowersOfTwo is O(log n). This bound meets our definition of simplicity. However, there is another, more precise way of stating an upper bound on the running time of PowersOfTwo, which is to say that it is big-oh of the function f (n) = m(n) + 1, where m(n) is the number of times 2 divides n. This function is hardly simple, as Fig. 3.6 shows. It oscillates wildly but never goes above 1 + log2 n. Since the running time of PowersOfTwo is O f (n) , but log n is not O f (n) , we claim that log n is not a tight bound on the running time. On the other hand, f (n) is a tight bound, but it is not simple. ✦
The Summation Rule Suppose a program consists of two parts, one of which takes O(n2 ) time and the other of which takes O(n3 ) time. We can “add” these two big-oh bounds to get the running time of the entire program. In many cases, such as this one, it is possible to “add” big-oh expressions by making use of the following summation rule: 4
Note that when we speak of logarithms within a big-oh expression, there is no need to specify the base. The reason is that if a and b are bases, then loga n = (logb n)(loga b). Since loga b is a constant, we see that loga n and logb n differ by only a constant factor. Thus, the functions logx n to different bases x are big-oh of each other, and we can, by the transitive law, replace within a big-oh expression any loga n by logb n where b is a base different from a.
106
THE RUNNING TIME OF PROGRAMS
Logarithms in Running Times If think of logarithms as something having to do with integral calculus (loge a = R ayou 1 dx), you may be surprised to find them appearing in analyses of algorithms. 1 x Computer scientists generally think of “log n” as meaning log2 n, rather than loge n or log10 n. Notice that log2 n is the number of times we have to divide n by 2 to get down to 1, or alternatively, the number of 2’s we must multiply together to reach n. You may easily check that n = 2k is the same as saying log2 n = k; just take logarithms to the base 2 of both sides. The function PowersOfTwo divides n by 2 as many times as it can, And when n is a power of 2, then the number of times n can be divided by 2 is log2 n. Logarithms arise quite frequently in the analysis of divide-and-conquer algorithms (e.g., merge sort) that divide their input into two equal, or nearly equal, parts at each stage. If we start with an input of size n, then the number of stages at which we can divide the input in half until pieces are of size 1 is log2 n, or, if n is not a power of 2, then the smallest integer greater than log2 n.
Suppose T1 (n) is known to be O f1 (n) , while T2 (n) is known to be O f2 (n). Further, suppose that f2 grows no faster than f1 ; that is, f2 (n) is O f1 (n) . Then we can conclude that T1 (n) + T2 (n) is O f1 (n) . In proof, we know that there are constants n1 , n2 , n3 , c1 , c2 , and c3 such that 1. 2. 3.
If n ≥ n1 , then T1 (n) ≤ c1 f1 (n). If n ≥ n2 , then T2 (n) ≤ c2 f2 (n). If n ≥ n3 , then f2 (n) ≤ c3 f1 (n).
Let n0 be the largest of n1 , n2 , and n3 , so that (1), (2), and (3) hold when n ≥ n0 . Thus, for n ≥ n0 , we have T1 (n) + T2 (n) ≤ c1 f1 (n) + c2 f2 (n) If we use (3) to provide an upper bound on f2 (n), we can get rid of f2 (n) altogether and conclude that T1 (n) + T2 (n) ≤ c1 f1 (n) + c2 c3 f1 (n) Therefore, for all n ≥ n0 we have T1 (n) + T2 (n) ≤ cf1 (n) if we define c to be c1 + c2 c3 . This statement is exactly what we need to conclude that T1 (n) + T2 (n) is O f1 (n) . ✦
Example 3.9. Consider the program fragment in Fig. 3.7. This program makes A an n × n identity matrix. Lines (2) through (4) place 0 in every cell of the n × n array, and then lines (5) and (6) place 1’s in all the diagonal positions from A[0][0] to A[n-1][n-1]. The result is an identity matrix A with the property that A×M = M×A = M
SEC. 3.5
(1) (2) (3) (4) (5) (6)
SIMPLIFYING BIG-OH EXPRESSIONS
107
scanf("%d",&n); for (i = 0; i < n; i++) for (j = 0; j < n; j++) A[i][j] = 0; for (i = 0; i < n; i++) A[i][i] = 1; Fig. 3.7. Program fragment to make A an identity matrix.
for any n × n matrix M. Line (1), which reads n, takes O(1) time, that is, some constant amount of time, independent of the value n. The assignment statement in line (6) also takes O(1) time, and we go around the loop of lines (5) and (6) exactly n times, for a total of O(n) time spent in that loop. Similarly, the assignment of line (4) takes O(1) time. We go around the loop of lines (3) and (4) n times, for a total of O(n) time. We go around the outer loop, lines (2) to (4), n times, taking O(n) time per iteration, for a total of O(n2 ) time. Thus, the time taken by the program in Fig. 3.7 is O(1) + O(n2 ) + O(n), for the statement (1), the loop of lines (2) to (4), and the loop of lines (5) and (6), respectively. More formally, if T1 (n) is the time taken by line (1), T2 (n) is the time taken by lines (2) to (4), and T3 (n) is the time taken by lines (5) and (6), then T1 (n) is O(1), T2 (n) is O(n2 ), and T3 (n) is O(n). We thus need an upper bound on T1 (n) + T2 (n) + T3 (n) to derive the running time of the entire program. Since the constant 1 is certainly O(n2 ), we can apply the rule for sums to conclude that T1 (n) + T2 (n) is O(n2 ). Then, since n is O(n2 ), we can apply the rule of sums to (T1 (n) + T2 (n)) and T3 (n), to conclude that T1 (n) + T2 (n) + T3 (n) is O(n2 ). That is, the entire program fragment of Fig. 3.7 has running time O(n2 ). Informally, it spends almost all its time in the loop of lines (2) through (4), as we might expect, simply from the fact that, for large n, the area of the matrix, n2 , is much larger than its diagonal, which consists of n cells. ✦ Example 3.9 is just an application of the rule that low order terms don’t matter, since we threw away the terms 1 and n, which are lower-degree polynomials than n2 . However, the rule of sums allows us to do more than just throw away low-order terms. If we have any constant number of terms that are, to within big-oh, the same, such as a sequence of 10 assignment statements, each of which takes O(1) time, then we can “add” ten O(1)’s to get O(1). Less formally, the sum of 10 constants is still a constant. To see why, note that 1 is O(1), so that any of the ten O(1)’s, can be “added” to any other to get O(1) as a result. We keep combining terms until only one O(1) is left.
108
THE RUNNING TIME OF PROGRAMS
However, we must be careful not to confuse “a constant number” of some term like O(1) with a number of these terms that varies with the input size. For example, we might be tempted to observe that it takes O(1) time to go once around the loop of lines (5) and (6) of Fig. 3.7. The number of times we go around the loop is n, so that the running time for lines (5) and (6) together is O(1) + O(1) + O(1) + · · · (n times). The rule for sums tells us that the sum of two O(1)’s is O(1), and by induction we can show that the sum of any constant number of O(1)’s is O(1). However, in this program, n is not a constant; it varies with the input size. Thus, no one sequence of applications of the sum rule tells us that n O(1)’s has any value in particular. Of course, if we think about the matter, we know that the sum of n c’s, where c is some constant, is cn, a function that is O(n), and that is the true running time of lines (5) and (6).
Incommensurate Functions It would be nice if any two functions f (n) and g(n) could be compared by big-oh; that is, either f (n) is O g(n) , or g(n) is O f (n) (or both, since as we observed, there are functions such as 2n2 and n2 + 3n that are each big-oh of the other). Unfortunately, there are pairs of incommensurate functions, neither of which is big-oh of the other.
✦
Example 3.10. Consider the function f (n) that is n for odd n and n2 for even n. That is, f (1) = 1, f (2) = 4, f (3) = 3, f (4) = 16, f (5) = 5, and so on. Similarly, let g(n) be n2 for odd n and let g(n) be n for even n. Then f (n) cannot be O g(n) , because of the even n’s. For as we observed in Section 3.4, n2 is definitely not O(n). Similarly, g(n) cannot be O f (n) , because of the odd n’s, where the values of g outrace the corresponding values of f . ✦
EXERCISES 3.5.1: Prove the following: na is O(nb ) if a ≤ b. na is not O(nb ) if a > b. an is O(bn ) if 1 < a ≤ b. an is not O(bn ) if 1 < b < a. na is O(bn ) for any a, and for any b > 1. an is not O(nb ) for any b, and for any a > 1. (log n)a is O(nb ) forany a, and for any b > 0. na is not O (log n)b for any b, and for any a > 0. 3.5.2: Show that f (n) + g(n) is O max f (n), g(n) . 3.5.3: Suppose that T (n) is O f (n) and g(n) is a function whose value is never negative. Prove that g(n)T (n) is O g(n)f (n) . 3.5.4: Suppose that S(n) is O f (n) and T (n) is O g(n) . Assume that none of these functions is negative for any n. Prove that S(n)T (n) is O f (n)g(n) . 3.5.5: Suppose that f (n) is O g(n) . Show that max f (n), g(n) is O g(n) .
a) b) c) d) e) f) g) h)
SEC. 3.6
ANALYZING THE RUNNING TIME OF A PROGRAM
109
3.5.6*: Show that if f1 (n) and f2 (n) are both tight bounds on some function T (n), then f1 (n) and f2 (n) are each big-oh of the other. 3.5.7*: Show that log2 n is not O f (n) , where f (n) is the function from Fig. 3.6. 3.5.8: In the program of Fig. 3.7, we created an identity matrix by first putting 0’s everywhere and then putting 1’s along the diagonal. It might seem that a faster way to do the job is to replace line (4) by a test that asks if i = j, putting 1 in A[i][j] if so and 0 if not. We can then eliminate lines (5) and (6). a)
Write this program.
b)* Consider the programs of Fig. 3.7 and your answer to (a). Making simplifying assumptions like those of Example 3.1, compute the number of time units taken by each of the programs. Which is faster? Run the two programs on varioussized arrays and plot their running times.
✦ ✦ ✦ ✦
3.6
Analyzing the Running Time of a Program Armed with the concept of big-oh and the rules from Sections 3.4 and 3.5 for manipulating big-oh expressions, we shall now learn how to derive big-oh upper bounds on the running times of typical programs. Whenever possible, we shall look for simple and tight big-oh bounds. In this section and the next, we shall consider only programs without function calls (other than library functions such as printf), leaving the matter of function calls to Sections 3.8 and beyond. We do not expect to be able to analyze arbitrary programs, since questions about running time are as hard as any in mathematics. On the other hand, we can discover the running time of most programs encountered in practice, once we learn some simple rules.
The Running Time of Simple Statements We ask the reader to accept the principle that certain simple operations on data can be done in O(1) time, that is, in time that is independent of the size of the input. These primitive operations in C consist of 1.
Arithmetic operations (e.g. + or %).
2.
Logical operations (e.g., &&).
3.
Comparison operations (e.g., <=).
4.
Structure accessing operations (e.g. array-indexing like A[i], or pointer following with the -> operator).
5.
Simple assignment such as copying a value into a variable.
6.
Calls to library functions (e.g., scanf, printf).
110
THE RUNNING TIME OF PROGRAMS
The justification for this principle requires a detailed study of the machine instructions (primitive steps) of a typical computer. Let us simply observe that each of the described operations can be done with some small number of machine instructions; often only one or two instructions are needed. As a consequence, several kinds of statements in C can be executed in O(1) time, that is, in some constant amount of time independent of input. These simple statements include
Simple statement
1.
Assignment statements that do not involve function calls in their expressions.
2.
Read statements.
3.
Write statements that do not require function calls to evaluate arguments.
4.
The jump statements break, continue, goto, and return expression, where expression does not contain a function call.
In (1) through (3), the statements each consist of some finite number of primitive operations, each of which we take on faith to require O(1) time. The summation rule then tells us that the entire statements take O(1) time. Of course, the constants hidden in the big-oh are larger for statements than for single primitive operations, but as we already know, we cannot associate concrete constants with running time of C statements anyway.
✦
Example 3.11. We observed in Example 3.9 that the read statement of line (1) of Fig. 3.7 and the assignments of lines (4) and (6) each take O(1) time. For another illustration, consider the fragment of the selection-sort program shown in Fig. 3.8. The assignments of lines (2), (5), (6), (7), and (8) each take O(1) time. ✦
(1) (2) (3) (4) (5) (6) (7) (8)
for (i = 0; i < n-1; i++) { small = i; for (j = i+1; j < n; j++) if (A[j] < A[small]) small = j; temp = A[small]; A[small] = A[i]; A[i] = temp; } Fig. 3.8. Selection-sort fragment.
Blocks of simple statements
Frequently, we find a block of simple statements that are executed consecutively. If the running time of each of these statements is O(1), then the entire block takes O(1) time, by the summation rule. That is, any constant number of O(1)’s sums to O(1).
SEC. 3.6
✦
ANALYZING THE RUNNING TIME OF A PROGRAM
111
Example 3.12. Lines (6) through (8) of Fig. 3.8 form a block, since they are always executed consecutively. Since each takes O(1) time, the block of lines (6) to (8) takes O(1) time. Note that we should not include line (5) in the block, since it is part of the if-statement on line (4). That is, sometimes lines (6) to (8) are executed without executing line (5). ✦
The Running Time of Simple For-Loops In C, many for-loops are formed by initializing an index variable to some value and incrementing that variable by 1 each time around the loop. The for-loop ends when the index reaches some limit. For instance, the for-loop of line (1) of Fig. 3.8 uses index variable i. It increments i by 1 each time around the loop, and the iterations stop when i reaches n − 1. There are more complex for-loops in C that behave more like while-statements; these loops iterate an unpredictable number of times. We shall take up this sort of loop later in the section. However, for the moment, focus on the simple form of for-loop, where the difference between the final and initial values, divided by the amount by which the index variable is incremented tells us how many times we go around the loop. That count is exact, unless there are ways to exit the loop via a jump statement; it is an upper bound on the number of iterations in any case. For instance, the for-loop of line (1) of Fig. 3.8 iterates (n − 1) − 0 /1 = n − 1 times, since 0 is the initial value of i, n − 1 is the highest value reached by i (i.e., when i reaches n − 1, the loop stops and no iteration occurs with i = n − 1), and 1 is added to i at each iteration of the loop. To bound the running time of the for-loop, we must obtain an upper bound on the amount of time spent in one iteration of the loop body. Note that the time for an iteration includes the time to increment the loop index (e.g., the increment statement i++ in line (1) of Fig. 3.8), which is O(1), and the time to compare the loop index with the upper limit (e.g., the test statement i
✦
Example 3.13. Consider the for-loop of lines (3) and (4) in Fig. 3.7, which is (3) (4)
for (j = 0; j < n; j++) A[i][j] = 0;
We know that line (4) takes O(1) time. Clearly, we go around the loop n times, as we can determine by subtracting the lower limit from the upper limit found on line (3) and then adding 1. Since the body, line (4), takes O(1) time, we can neglect the time to increment j and the time to compare j with n, both of which are also O(1). Thus, the running time of lines (3) and (4) is the product of n and O(1), which is
112
THE RUNNING TIME OF PROGRAMS
O(n). Similarly, we can bound the running time of the outer loop consisting of lines (2) through (4), which is (2) (3) (4)
for (i = 0; i < n; i++) for (j = 0; j < n; j++) A[i][j] = 0;
We have already established that the loop of lines (3) and (4) takes O(n) time. Thus, we can neglect the O(1) time to increment i and to test whether i < n in each iteration, concluding that each iteration of the outer loop takes O(n) time. The initialization i = 0 of the outer loop and the (n + 1)st test of the condition i < n likewise take O(1) time and can be neglected. Finally, we observe that we go around the outer loop n times, taking O(n) time for each iteration, giving a total O(n2 ) running time. ✦
✦
Example 3.14. Now, let us consider the for-loop of lines (3) to (5) of Fig. 3.8. Here, the body is an if-statement, a construct we shall learn how to analyze next. It is not hard to deduce that line (4) takes O(1) time to perform the test and line (5), if we execute it, takes O(1) time because it is an assignment with no function calls. Thus, we take O(1) time to execute the body of the for-loop, regardless of whether line (5) is executed. The incrementation and test in the loop add O(1) time, so that the total time for one iteration of the loop is just O(1). Now we must calculate the number of times we go around the loop. The number of iterations is not related to n, the size of the input. Rather, the formula “last value minus initial value divided by the increment” gives us n − (i + 1) /1, or n − i − 1, as the number of times around the loop. Strictly speaking, that formula holds only if i < n. Fortunately, we can observe from line (1) of Fig. 3.8 that we do not enter the loop body of lines (2) through (8) unless i ≤ n−2. Thus, not only is n−i−1 the number of iterations, but we also know that this number cannot be 0. We conclude that the time spent in the loop is (n − i − 1) × O(1), or O(n − i − 1).5 We do not have to add in O(1) for initializing j, since we have established that n − i − 1 cannot be 0. If we had not observed that n − i − 1 was positive, then we would have to write the upper bound on the running time as O max(1, n − i − 1) . ✦ The Running Time of Selection Statements An if-else selection statement has the form if () else <else-part> where 1.
The condition is an expression to be evaluated, 5
Technically, we have not discussed a big-oh operator applied to a function of more than one variable. In this case, we can regard O(n − i − 1) as saying “at most some constant times n − i − 1.” That is, we can consider n − i − 1 as a surrogate for a single variable.
SEC. 3.6
ANALYZING THE RUNNING TIME OF A PROGRAM
113
2.
The if-part is a statement that is executed only if the condition is true (the value of the expression is not zero), and
3.
The else-part is a statement that is executed if the condition is false (evaluates to 0). The else followed by the <else-part> is optional.
A condition, no matter how complex, requires the computer to perform only a constant number of primitive operations, as long as there are no function calls within the condition. Thus, the evaluation of the condition will take O(1) time. Suppose that there are no function calls in the condition, and that the if- and else-parts have big-oh upper bounds f (n) and g(n), respectively. Let us also suppose that f (n) and g(n) are not both 0; that is, while the else-part may be missing, the if-part is something other than an empty block. We leave as an exercise the question of determining what happens if both parts are missing or are empty blocks. If f (n) is O g(n) , then we can take O g(n) to be an upper bound on the running time of the selection statement. The reason is that 1.
We can neglect the O(1) for the condition,
2.
If the else-part is executed, g(n) is known to be a bound on the running time, and
3.
If the if-part is executed instead of the else-part, the running time will be O g(n) because f (n) is O g(n) . Similarly, if g(n)is O f (n) , we can bound the running time of the selection statement by O f (n) . Note that when the else-part is missing, as it often is, g(n) is 0, which is surely O f (n) . The problem case is when neither f nor g is big-oh of the other. We know that either the if-part or the else-part, but not both, will be executed, and so a safe upper bound on the running time is the larger of f (n) and g(n). Which is larger can depend on n, as we saw in Example 3.10. Thus, we must write the running time of the selection statement as O max f (n), g(n) .
✦
Example 3.15. As we observed in Example 3.12, the selection statement of lines (4) and (5) of Fig. 3.8 has an if-part, line (5), which takes O(1) time, and a missingelse-part, which takes 0 time. Thus, f (n) is 1 and g(n) is 0. As g(n) is O f (n) , we get O(1) as an upper bound on running time for lines (4) and (5). Note that the O(1) time to perform the test A[j] < A[small] at line (4) can be neglected. ✦
✦
Example 3.16. For a more complicated example, consider the fragment of code in Fig. 3.9, which performs the (relatively pointless) task of either zeroing the matrix A or setting its diagonal to 1’s. As we learned in Example 3.13, the running time of lines (2) through (4) is O(n2 ), while the running time of lines (5) and (6) is O(n). Thus, f (n) is n2 here, and g(n) is n. Since n is O(n2 ), we can neglect the time of the else-part and take O(n2 ) as a bound on the running time of the entire fragment of Fig. 3.9. That is to say, we have no idea if or when the condition of line (1) will be true, but the only safe upper bound results from assuming the worst: that the condition is true and the if-part is executed. ✦
114
THE RUNNING TIME OF PROGRAMS
(1) (2) (3) (4)
if (A[1][1] == 0) for (i = 0; i < n; i++) for (j = 0; j < n; j++) A[i][j] = 0; else for (i = 0; i < n; i++) A[i][i] = 1;
(5) (6)
Fig. 3.9. Example of an if-else selection statement.
The Running Time of Blocks We already mentioned that a sequence of assignments, reads, and writes, each of which takes O(1) time, together takes O(1) time. More generally, we must be able to combine sequences of statements, some of which are compound statements, that is, selection statements or loops. Such a sequence of simple and compound statements is called a block. The running time of a block is calculated by taking the sum of the big-oh upper bounds for each of the (possibly compound) statements in the block. With luck, we can use the summation rule to eliminate all but one of the terms in the sum.
Compound statement
✦
Example 3.17. In the selection sort fragment of Fig. 3.8, we can view the body of the outer loop, that is, lines (2) through (8), as a block. This block consists of five statements: 1. 2. 3. 4. 5.
The The The The The
assignment of line (2) loop of lines (3), (4), and (5) assignment of line (6) assignment of line (7) assignment of line (8)
Note that the selection statement of lines (4) and (5), and the assignment of line (5), are not visible at the level of this block; they are hidden within a larger statement, the for-loop of lines (3) to (5). We know that the four assignment statements take O(1) time each. In Example 3.14 we learned that the running time of the second statement of the block — that is, lines (3) through (5) — is O(n − i − 1). Thus, the running time of the block is O(1) + O(n − i − 1) + O(1) + O(1) + O(1) Since 1 is O(n − i − 1) (recall we also deduced that i never gets higher than n − 2), we can eliminate all the O(1)’s by the summation rule. Thus, the entire block takes O(n − i − 1) time. For a