Theory and Applications of Numerical Analysis

Eastbourne

PREFACE

Although no text of reasonable length can cover all possible topics, it seemed to us that the first edition of this text would be improved by including material on two further items. Thus the main change in the second edition is the addition of two entirely new chapters, Chapter 6 (Splines and other approximations) and Chapter 11 (Matrix eigenvalues and eigenvectors). In the preface to the first edition we stated that the material was equivalent to about sixty lectures. This new edition has adequate material for two separate semester courses.

When this book first appeared in 1973, computers were much less ‘friendly’ than they are now (1995) and, in general, it was only specialists who had access to them. The only calculating aids available to the majority of our readers were mathematical tables and desk calculators; the latter were mainly mechanical machines, which are now museum pieces. In contrast, the readership of our second edition will be able to appreciate the power and elegance of the algorithms which we discuss by implementing them, and experimenting with them, on the computer. (Sounding like our parents of old, we can say to our students ‘We never had it so easy: you are lucky!’) To encourage the active pursuit of the algorithms in the text, we have included computing exercises amongst the problems at the end of each chapter.

Despite the changes made in the second edition, the flavour of this text remains the same, reflecting the authors’ tastes: in short, we like both theorems and algorithms and we remain in awe of the masters of our craft who discovered or created them. The theorems show how the mathematics hangs together and point the way to the algorithms.

We are deeply grateful to our readership and to our publishers for keeping this text in print for such a long time. We also owe much to the many colleagues from many countries with whom we have shared the sheer fun of studying mathematics in our research collaborations, and we mention their names here as a sign of our respect and gratitude: G. E. Bell, L. Brutman, B. L. Chalmers, E. W. Cheney, F. Deutsch, D. K. Dimitrov, D. Elliott, Feng Shun-xi, D. M. E. Foster, H. T. Freitag, R. E. Grundy, A. S. B. Holland, Z. F. Koç ak, S. L. Lee, W. Light, J. H. McCabe, A R. Mitchell, D. F. Paget, A. Sri Ranga, B. N. Sahney, S. P. Singh, E. L. Wachspress, M. A. Wolfe, D. Yahaya.

G.M. PHILLIPS,     Mathematical Institute, University of St Andrews, Scotland

P.J. TAYLOR,     formerly of the Department of Mathematics, University of Strathclyde, Scotland

December 1995

FROM THE PREFACE TO THE FIRST EDITION

We have written this book as an introductory text on numerical analysis for undergraduate mathematicians, computer scientists, engineers and other scientists. The material is equivalent to about sixty lectures, to be taken after a first year calculus course, although some calculus is included in the early chapters both to refresh the reader’s memory and to provide a foundation on which we may build. We do not assume that the reader has a knowledge of matrix algebra and so have included a brief introduction to matrices. It would, however, help the understanding of the reader if he has taken a basic course in matrix algebra or is taking one concurrently with any course based on this book.

We have tried to give a logical, self-contained development of our subject ab initio, with equal emphasis on practical methods and mathematical theory. Thus we have stated algorithms precisely and have usually given proofs of theorems, since each of these aspects of the subject illuminates the other. We believe that numerical analysis can be invaluable in the teaching of mathematics. Numerical analysis is well motivated and uses many important mathematical concepts. Where possible we have used the theme of approximation to give a unified treatment throughout the text. Thus the different types of approximation introduced are used in the chapters on the solution of non-linear algebraic equations, numerical integration and differential equations.

It is not easy to be a good numerical analyst since, as in other branches of mathematics, it takes considerable skill and experience to be able to leap nimbly to and fro from the general and abstract to the particular and practical. A large number of worked examples has been included to help the reader develop these skills. The problems, which are given at the end of each chapter, are to be regarded as an extension of the text as well as a test of the reader’s understanding.

Some of our colleagues have most kindly read all or part of the manuscript and offered wise and valuable advice. These include Dr J. D. Lambert, Prof. P. Lancaster, Dr J. H. McCabe and Prof. A. R. Mitchell. Our thanks go to them and, of course, any errors or omissions which remain are entirely our responsibility We are also much indebted to our students and to our colleagues for the stimulus given by their encouragement.

December 1972

G.M. PHILLIPS,     Department of Applied Mathematics, University of St Andrews, Scotland

P.J. TAYLOR,     Department of Computing Science, University of Stirling, Scotland

Chapter 1

INTRODUCTION

1.1 What is numerical analysis

Numerical analysis is concerned with the mathematical derivation, description and analysis of methods of obtaining numerical solutions of mathematical problems. We shall be interested in constructive methods in mathematics; these are methods which show how to construct solutions of mathematical problems. For example, a constructive proof of the existence of a solution to a problem not only shows that the solution exists but also describes how a solution may be determined. A proof which shows the existence of a solution by reductio ad absurdum† is not constructive. Consider the following simple example.

Example 1.1

Prove that the quadratic equation

(1.1)

with real coefficients b and c satisfying b² > c has at least two real roots.

Constructive proof

For all x, b and c,

Thus x is a root of (1.1) if and only if

that is,

On taking square roots, remembering that b² − c > 0, we see that x is a root if and only if

that is

(1.2)

Thus there are two real roots and (1.2) shows how to compute these.

Non-constructive proof

Let

Suppose first that there are no roots of (1.1) and hence no zeros of q. As q is a continuous function (see § 2.2), we see that q(x) is either always positive or always negative for all x. Now

by the condition on b and c. Thus q(x) must always be negative. However, for |x| large,

and thus q(x) > 0 for |x| large. We have a contradiction.

Secondly, suppose that q has only one zero. Then from the continuity of q we must have q(x) > 0 for x large and q(x) < for − x large or vice versa. This contradicts the fact that q(x) > 0 for |x| large, regardless of the sign of x

The word ‘algorithm’ has for a long time been used as a synonym for ‘method’ in the context of constructing the solution of a mathematical problem. Measured against the long history of mathematics it is only relatively recently that a proper mathematical definition of algorithm has been devised. Much of this important work is due to A. M. Turing (1912–54), who gave a definition based on an abstract concept of a computer. For the purposes of this book we define an algorithm to be a complete and unambiguous description of a method of constructing the solution of a mathematical problem. One difficulty in making a formal definition is deciding precisely what operations are allowed in the method. In this text we shall take these to be the basic arithmetical operations of addition, subtraction, multiplication and division. In general, the solution and method occurring in an algorithm need not be numerical. For example, the early Greek geometers devised algorithms using ruler, pencil and compass. One such algorithm is that for constructing the perpendicular to a straight line at a given point. However, the greatest of the Greek mathematicians, Archimedes (287–212 BC), constructed a most elegant numerical algorithm for approximating the area of a circle (thus estimating π), by bounding it above and below by the areas of regular polygons with an increasingly large number of sides. In this book we shall deal only with numerical algorithms.

It is interesting to note that until the beginning of the twentieth century much mathematical analysis was based on constructive methods, and some of the finest mathematicians of every era have produced important algorithms and constructive proofs, for example Isaac Newton (1642–1727), L. Euler (1707–83) and K. F. Gauss (1777–1855). It is perhaps ironic that the trend in more recent years has been towards non-constructive methods, although the introduction of calculating devices made the execution of algorithms much easier.

The rapid advances in the design of digital computers have, of course, had a profound effect on numerical analysis. At the time the first edition of this book was published in 1973, computers could already perform over a million arithmetic operations in a second, while around 1960 speeds had been nearer one thousand operations a second. As we prepare the second edition of this text we reflect that the most impressive and important development in computers over the past two decades has been the dramatic decrease in their cost, leading to their near universal availability. Yet, while lengthier and more complex calculations may be tackled, the scope for error has correspondingly increased. Unfortunately, the expectations of users of numerical algorithms have more than matched the improvements in computers. However, one cannot select the best algorithm for the problem in hand without having a sound understanding of the theoretical principles involved.

Although most numerical algorithms are designed for use on digital computers, the subject of numerical analysis should not be confused with computer programming and information processing (or data processing). Those who design and use numerical algorithms need a knowledge of the efficient use of a digital computer for performing calculations. Computer programming is primarily concerned with the problem of coding algorithms (not necessarily numerical) in a form suitable for a computer. Information processing is concerned with the problems of organizing a computer so that data can be manipulated. The data is not necessarily numerical.

1.2 Numerical algorithms

For many problems there are no algorithms for obtaining exact numerical values of solutions and we have to be satisfied with approximations to the values we seek. The simplest example is when the result is not a rational number and our arithmetic is restricted to rational numbers. What we do ask of an algorithm, however, is that the error in the result can be made as small as we please. Usually, the higher the accuracy we demand, the greater is the amount of computation required. This is illustrated in the following algorithm† for computing a square root, based on bisection, where a demand for higher accuracy in the result means completing more basic arithmetical operations, as well as increasing the accuracy with which these are performed.

Example 1.2

Given a with error not greater than ε. At each stage we have two numbers x0 and x1. In the calculation, we change x0 and xand at the same time repeatedly halving the interval [x0, x1]. Initially we take x0 = 1 and x1 = aWe compute the midpoint x = (x0 + x1)/2 and then decide whether x we compare x² and a. If x² > a we replace the number x1 by x; otherwise we replace the number x0 by xwhere ε was chosen in advance to give the accuracy we desire in the final approximation x.

Algorithm 1.1

(square root by bisection) We begin with a > 1 and ε > 0, and x0 = 1, x1 = a.

In the above algorithm, the symbol := should be interpreted as ‘becomes equal to’. The ‘loop’ of instructions between repeat and until is called an iteration. If we make the positive number ε smaller, the number of iterations required increases. We must also make the arithmetic more accurate as ε is decreased, as otherwise the test whether x² > a will give incorrect results and x0 and x

In Algorithm 1.1 we compute the first few members of a sequence of values (the midpoints x) Many algorithms consist of calculating a sequence and in this context we say that the algorithm is convergent if the sequence converges to the desired solution of the problem.

In the analysis of an algorithm we shall consider the error in the computed approximation to the solution of our problem. This error may be due to various factors, for example, rounding in arithmetic and the termination of an infinite process, as in Example 1.2. Errors, which are a deliberate but often unavoidable part of the solution process, should not be confused with mistakes which are due to failure on the part of the human user of an algorithm, particularly when writing a computer program. For each algorithm we look for an error bound, that is, a bound on the error in the computed solution. A bound which is not directly dependent on the solution and therefore can be computed in advance, is called an a priori error bound. When directly dependent on the solution, a bound cannot be computed until after the completion of the main part of the calculation and is called an a posteriori bound. If, as is often the case, an error bound is very much larger than the actual error, the bound cannot be used to provide a sensible estimate of the error. For this reason we also devise alternative methods of estimating the error for some problems. Error bounds and estimates are often very valuable in telling us about the asymptotic behaviour of the error as we improve approximations made in deriving an algorithm.

There are two ways in which we measure the magnitude of an error. If a is some real number and a* is an approximation to a we define the absolute error to be | a − a* | and we define the relative error We can see that the absolute error may be quite meaningless unless we have some knowledge of the magnitude of a. We often give relative errors in terms of percentages.

One important point we must consider in looking at an algorithm is its efficiency. We usually measure this in terms of the number of basic arithmetical operations required for a given accuracy in the result. Sometimes we also consider other factors such as the amount of computer storage space required.

1.3 Properly posed and well-conditioned problems

There is one important question which we must investigate before applying a numerical method to a problem and that is ‘How sensitive is the solution to changes in data in the formulation of the problem?’ For example, if we are solving a set of algebraic equations, the coefficients in the equations form data which must be provided. There will certainly be difficulties if small changes in these coefficients have a large effect on the solution. It is very unlikely that we can compute a solution without perturbing these coefficients if only because of errors due to rounding in the arithmetic.

Suppose that S(d) represents† a solution of a problem for a given set of data d. Now suppose that d + δd is a perturbed set of data and S(d + δd) δd S(d + δd) − S(d) to denote a non-negative number which is a measure of the magnitude of the difference in the solutions. We say that the problem is properly posed for a given set of data d if the following two conditions are satisfied.

(i) A unique solution exists for the set of data d and for each set of data ‘near’ d. That is, there is a real number ε > 0 such that S(d + δd) exists and is unique for all δd δd < ε.

(ii) The solution S(d) is continuously dependent on the data at d, that is

If there were more than one solution for a given set of data, then we would not know which solution (if any) our numerical method obtains. Usually we can avoid existence of more than one solution by imposing extra conditions on the problem. For example, if we wish to solve a cubic equation we could stipulate that the largest real root is required.

If the condition (ii) above is not satisfied, it would be difficult to use a numerical algorithm as then there are data which are arbitrarily close to d and such that the corresponding solutions are quite different from the desired solution S(d).

We say that a problem which is properly posed for a given set of data is well-conditioned if every small perturbation of that data results in a relatively small change in the solution. If the change in the solution is large we say that the problem is ill-conditioned. Of course these are relative terms.

Often the solution of a properly posed problem is Lipschitz dependent on the data (see § 2.2). In this case there are constants L > 0 and ε > 0 such that

(1.3)

for all δd δd < ε. In this case we can see that the problem is well-conditioned if L δd < ε, only means that there are limits within which the data must lie if (1.3) is to hold. This restriction should not worry us unless ε is small. If L has to be made large before (1.3) is satisfied or if no such L exists, then a small change in the data can result in a large change in the solution and thus the problem is ill-conditioned.

As we remarked earlier, an ill-conditioned problem may be difficult to solve numerically as rounding errors may seriously affect the solution. In such a case we would at least need to use higher accuracy in the arithmetic than is usual. If the problem is not properly posed, rounding errors may make the numerical solution meaningless, regardless of the accuracy of the arithmetic. Sometimes the data contains inherent errors and is only known to a certain degree of accuracy. In such cases the amount of confidence we can place in the results depends on the conditioning of the problem. If it is ill-conditioned, the results may be subject to large errors and, if the problem is not properly posed, the results may be meaningless.

The next example illustrates that the properties properly posed and well-conditioned depend on both the problem and the data.

Example 1.3

We consider the computation of values of the quadratic

This is an ill-conditioned problem if x is near a root of the quadratic. For example, for x = 100/3, q−5.6. However, if we take x = 33, which differs from 100/3 by only 1%, we find that q(33)= −28 which is approximately five times the value of q(100/3). To obtain a relation of the form (1.3) when x = 100/3 we need to take L

Suppose that we wish to solve a problem which we will denote by P. We often find it necessary to replace P by another problem P1, (say) which approximates to P and is more amenable to numerical computation. We now have two types of error. Firstly, there is the difference between P and P1 and we write EP to denote some measure of this error. Often this results from cutting off an infinite process such as forming the sum of an infinite series (see § 2.3) after only a finite number of steps, in which case we call it the truncation error. Secondly, there is the error consisting of the difference between the solutions of P and P1 and we denote some measure of this by ES. This second type of error is sometimes called the global error. Of course there is a relation between EP and ES, but it does not follow that if EP is small then so is ES. We shall derive relations between EP and ES for different types of problem and, for a satisfactory method, will require ES to become arbitrarily small as the approximation is improved, so that EP tends to zero. If this property does hold we say that the resulting algorithm is convergent. Unfortunately there are problems and approximations such that ES does not tend to zero as EP is decreased.

Example 1.4

Suppose that f is a given real-valued differentiable function of a real variable x and that we seek values of the derivative f′ for given particular values of x in the interval a ≤ x ≤ b. We will assume that f’ cannot be obtained explicitly as f is a very complicated function but that we can compute f(x) for any given value of x.

Our algorithm consists of replacing f by another function g which can easily be differentiated. We then compute values of g′(x) instead of f′(x). In Chapter 5 we will describe one way in which g can be chosen and that provided f satisfies certain conditions we can make the error EP(x) = f(x) − g(x) arbitrarily small for all x with a ≤ x ≤ b. Unfortunately it is possible for Es(x) = f′(x) − g′(x) to be arbitrarily large although EP(x) is arbitrarily small. For example, if EP(x) = (1/n) sin(n²x), where n = 1, 2, 3, … corresponds to different choices of g, then

for all values of x. However,

and thus Es(x) → ∞ as n → ∞ for x

If we replace a problem P by another, more convenient problem P1 which is an approximation to P, we shall normally expect both P and P1 to be properly posed and reasonably well-conditioned. If P were not properly posed, in general we would not obtain convergence of the solution of P1 to that of P. We are almost certain in effect to perturb the data of P in making the approximation P1. We therefore require, as a necessary condition for the convergence of our algorithm, that the original problem P be properly posed. The convergence will be ’slow’ unless P is well-conditioned.

We also have to solve P1 using numerical calculations and we are almost certain to perturb the data of P1, which should therefore be properly posed and well-conditioned. If P is well-conditioned, we normally expect P1 to be well-conditioned; if not, we should look for a better alternative than P1.

In some numerical algorithms, especially those for differential equations, we compute elements of a sequence as in Example 1.2 but are interested in each element of the sequence and not just the limit. In both types of problem we use a recurrence relationusing a recurrence relation of the form

(1.4)

for n = m, m + 1, m + 2, …, where F is a known function of m + 1 variables. We need starting values y0, y0, …, ym and normally these must be given. In using (1.4) we are certain to make rounding errors. Now an error introduced in any one step (one value of n) will affect all subsequent values and, therefore, we need to investigate the propagated effect of rounding errors. These effects may be very important, especially as computers allow us to attempt to compute very many elements of a sequence and so introduce many rounding errors. In this context we use the term stability in referring to whether the process is properly posed and well-conditioned.

Example 1.5

satisfies

(1.5)

with y0 = l and y1 = 0.01 then y5 = 10−10, whereas if y0 = l + 10−6 and y1 = 0.01 then y−1.0001. The large difference in the value of y5 shows that the process is very ill-conditioned. In fact we say that the recurrence relation is numerically unstable. Further details are given in

Problems

Section 1.1

1.1. Give both constructive and non-constructive proofs of the following statements.

(i) Every quadratic equation has at most two distinct real roots.

(ii) There are at least two real values of x such that

and a point P which passes through P.

Section 1.2

1.2. Show that an a priori with 0 < a < 1 is

where N is the number of iterations completed.

1.3. In Algorithm 1.1 we have at each stage

Show that if

then the relative error in x satisfies

Hence modify the algorithm so that the iterations stop when the relative error is less than ε.

Section 1.3

1.4. Show that the equations

have a unique solution, but that the problem of computing this solution is not properly posed, in the sense given in § 1.3. Notice that, if we express coefficients as rounded decimal fractions, there is no solution.

1.5. If q(x) is the quadratic of Example 1.3, show that

x 34 and | δ x 1, then

†In such a proof we assume that the solution does not exist and obtain a contradiction.

†This algorithm is not recommended; it is given merely as a simple illustration. A more efficient square root algorithm is described in § 8.6.

†S(d) and d may be numbers, functions, matrices, vectors or combinations of these depending on the problem.

Chapter 2

BASIC ANALYSIS

2.1 Functions

Definition 2.1

A set

We write

where S denotes a set and a, b, c, … denote the objects belonging to the set. We call a, b, c, … the elements or members of the set S and we write a ∈ S as a shorthand for ‘a belongs to S’. If S and T are sets such that†x ∈ T ⇒ x ∈ S, we say that T is a subset of S and write T ⊂ S.

Example 2.1

The set of all numbers x such that a x b is denoted by [a, b] and is called a closed interval. The set of x such that a < x < b is denoted by (a, b) and is called an open interval. We write (a, b] to denote the set of x such that a < x b

Definition 2.2

If, given two sets X and Y, there is a relation between them such that each member of X is related to exactly one member of Y, we call this relation a function or mapping from X to Y

Example 2.2

If, at some instant, X denotes the set of all London buses in use and Y the set of all human beings, the relation ‘y is the driver of x, with x ∈ X, y ∈ Y’ is a function from X to Y.