Numerical Analysis Chapter 2

Introduction
Bisection method
The fixed point method
Newtons method
Secant method
Other topics
Numerical Analysis and Computation

Chapter 2: Nonlinear Equations in One Variable
Weiwei Hu
Department of Mathematics
University of Southern Califronia
1 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
2 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
3 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Goal
Solve f (x) = 0 for a given function f (x). That is, find the
roots of f (x).
Why do we start from this problem?
How to guarantee that the given problem has a unique
solution on a specified domain?
How to construct a convergent numerical method?
How to obtain a faster convergence?
What if there are more than one roots?
..........
4 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Reasons to begin with f (x) = 0
This problem occurs frequently.

This problem provides us a quick but complete look at
different numerical methods for a problem.
Solving this problem illustrates the iterative method, which is
a common numerical technique.
Moreover, it illustrates that when many methods are available,
selection of the most suitable method depends on the specific
problem, the goal, and the computation equipments available.
5 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Existence of the solution
Theorem (Intermediate Value Theorem)

If f C [a, b] and is some number between f (a) and f (b), then
there exists (a, b) such that f () = .
6 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Existence of the solution
Corollary (I)
If f C [a, b] and f (a)f (b) < 0, then there is a (a, b) such
that f () = 0.
7 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Uniqueness of the solution
Corollary (II)
If f C [a, b], f is monotonic on [a, b] and f (a)f (b) < 0, then
there is a unique (a, b) such that f () = 0.
Remark
f 0 (x) can provide the monotonicity of f (x) on certain
intervals.
If there are more than one solutions in [a, b], then the
numerical methods may or may not converge!
8 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
9 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Basic idea of the bisection method

Suppose that f (x) = 0 has only one solution z [a, b], then
there are three possibilities for the location of z:
1
2
3
z [a, a+b
2 );
z ( a+b
2 , b];
z = a+b
2 .
10 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

1
2
3
z [a, a+b
2 );
z ( a+b
2 , b];
z = a+b
2 .
Once we identify the half interval which contains z, we define

it to be the new interval and its length is half of that of the
original interval [a, b].
10 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

1
2
3
z [a, a+b
2 );
z ( a+b
2 , b];
z = a+b
2 .

Then we repeat this procedure on the new interval again and
again to obtain smaller and smaller intervals which contains
the solution z.
10 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

1
2
3
z [a, a+b
2 );
z ( a+b
2 , b];
z = a+b
2 .

Then we repeat this procedure on the new interval again and
again to obtain smaller and smaller intervals which contains
the solution z.
When the interval is small enough, the procedure can be
stopped and any point in the interval is a good approximation
to the solution z. Usually we pick up the middle point of the
last interval to be the final numerical solution.
10 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Algorithm of the bisection method

1
Find a and b such that f (a)f (b) < 0 and f (x) is monotonic on [a, b].
Otherwise, stop.
Initial preparation:
(1) Define a0 = a and b0 = b.

(2) Set the iteration counter k = 0.
(3) Specify a tolerance to stop the following loop.
3
While
bk ak
2
, repeat the following procedure in a loop:
k
(1) xk = ak +b
2 .
(2) If f (xk ) = 0, then
stop the loop;
elseif f (ak )f (xk ) < 0, then
ak+1 = ak and bk+1 = xk ;
elseIf f (xk )f (bk ) < 0, then
ak+1 = xk and bk+1 = bk .
end
(3) k k + 1.
Output the final xk from the above loop.

11 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Matlab for the bisection method
Several implementation issues:

No need to store all of ak and bk . We may just define two
variables a and b, then store the current ak and bk into them.
We have two ways to code for a loop: for and while.
Example:
for k = 0 : 10
while (a < 10)
.....
.....
end
end
Use if... elseif...elseif...else...end structure instead of several
if...end structures.
12 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Stop Criteria
Interval [ak , bk ] is small enough;
|f (xk )| is almost 0;
Max number of iterations is reached;
Any combination of the previous ones.
13 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Example 1: f (x) = e x + x, f (0) = 1, and f (1) = 0.632.

Choose = 0.01.
f (x) is monotonically increasing on any interval since
f 0 (x) = e x + 1 > 0 for all x.
14 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Based on the code we developed together just now, we obtain
Numerical results:
k
0
1
2
3
4
5
6
ak
1
1
0.75
0.625
0.625
0.59375
0.578125
bk
0
0.5
0.5
0.5
0.5625
0.5625
0.5625
xk
0.5
0.75
0.625
0.5625
0.59375
0.578125
0.5703125
Verification: f (0.5703125) 4.963760 103 . A smaller

tolerance will give a smaller error.
15 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Convergence theory of the bisection method
Theorem (I)
Assume z is the only solution of f (x) = 0 on [a, b]. If f is a
continuous function on [a, b] and f (a)f (b) < 0, then
|xk z|
ba
.
2k+1
Proof.
z [ak , bk ] and |xk z|
Then combine this with bk ak =
ba
,
2k
b k ak
.
2
we finish the proof.
16 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Example 2: Assume the solution z of f (x) = 0 is in [0, 2].
How many iteration steps are required to reduce the error to
less than 104 ?
Solution:
|xk z|
ba
20
104 k+1 104 .
k+1
2
2
Then
2k 104 k
ln104
13.2877.
ln2
Hence 14 iteration steps are required.

17 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
The bisection method is limited since it is not valid for many cases.
For example, it is not valid for complex solutions of f (x) = 0.
Furthermore, if f (x) 0 or f (x) 0, then the bisection method is
not valid since f (a)f (b) < 0 requires either f (a) < 0 or f (b) < 0.
Remark
Although the convergence of the bisection method is not fast, the
bisection method is a good choice for finding a better real initial
point for other advanced iterative numerical methods.
18 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
19 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Basic idea of the fixed point method

Still consider f (x) = 0.
What if f (x) = g (x) x or f (x) = x g (x)?
Then f (x) = 0 x = g (x).
Define z to be a fixed point if z = g (z).
How about this iteration: xk+1 = g (xk )?
A geometric explanation of the fixed point method: see Figure
3.2 on page 47 of the textbook.
20 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Stop Criteria
Let and f be the tolerance

|xk+1 xk | < ;
|g (xk ) xk | < f ;
A maximum number M of iterations is reached;
Any combination.
21 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Algorithm of the fixed point method

1
(1) Pick up the initial point x0 .
(2) Specify the tolerances , f , and the maximum number M
of iteration steps.
First step: k = 0, x1 = g (x0 ).
While |xk+1 xk | and k M 1, repeat the following

procedure in a loop:
(1) xk+1 = g (xk ).
(2) If |g (xk+1 ) xk+1 | < f , then stop the loop.
(3) k k + 1.
Output the final xk+1 from the above loop as the the
numerical solution.
22 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Matlab for the fixed point method
Example 1: f (x) = x+1

and x0 = 0. Then g (x) =
2
6
Choose = 0.01, f = 10 , and M = 100. Its
straightforward to verify that the fixed point is 1.
x+1
2 .
23 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Practice in Matlab for the fixed point method

Based on the code we developed together just now, we obtain
Numerical results:
k
xk
0
0
1
0.5
2
0.75
3
0.875
4
0.9375
5
0.96875
6
0.984375
7 0.9921875
Verification:
f (0.9921875) = g (0.9921875) 0.9921875 0.00390625. A
smaller tolerance will give a smaller error.
Try other initial values? Similar performance.
24 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Example 2: f (x) = 4 + 13 sin 2x x and x0 = 0 . Then
g (x) = 4 + 31 sin 2x.
Numerical results:
k
0
1
2
3
4
5
xk
4
4.329786
4.230895
4.273634
4.273634
4.256383
Verification:
f (4.256383) = g (4.256383) 4.256383 0.00719599.
Try other initial values?
Similar performance.
25 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Example 3: f (x) = x2 + x1 . Then g (x) = x2 + x1 . Its
straightforward
to
verify
that
there
are
two
fixed
points:
2
and 2. Choose x0 = 10 and x0 = 5 separately.

Numerical results:
k xk from x0 = 10 xk from x0 = 5
0
10
5
1
5.1
2.7
2
2.746078
1.720370
3
1.737195
1.441455
4
1.444238
1.414471
5
1.414526
1.414214
6
1.414214
Verification: f (1.414214) 4.376268 107 and
f (1.414214) 4.376268 107 .
Try other initial values? Similar performance.
26 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Example 4: f (x) = x 2 x. Then g (x) = x 2 . Choose x0 = 2
and x0 = 0.5 separately.
Numerical results:
k
0
1
2
3
4
5
xk from x0 = 2
2
4
16
256
6.5536 104
4.2950 109
xk from x0 = 0.5
0.5
0.25
0.0625
3.90625 103
1.5259 105
Diverge sometimes!
27 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Example 5: f (x) = e x + x and x0 = 0.
or g (x) = e x .
Numerical results:
k
xk for g (x) = e x + 2x xk
0
0
1
1
2
4.718282
3
1.214123 102
4
5.3539 1052
5
6
7
8
9
10
Then g (x) = e x + 2x
for g (x) = e x
0
1
0.367879
0.692201
0.500474
0.606244
0.545396
0.579612
0.560115
0.571143
0.564879
28 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Verification for the case with g (x) = e x :

f (0.564879) 3.549922 103 .
Try other initial values for the case with g (x) = e x + 2x?
Diverge, diverge, diverge for g (x) = e x + 2x! Whats wrong?
29 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Question 1: When is there a fixed point of the original

problem?
Question 2: When is the solution unique?
Question 3: When do the numerical solutions {xk } converge
to that solution?
Question 4: How fast do the numerical solutions {xk }
converge to that solution?
30 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Question 1: When is there a fixed point of the original

problem?
Question 2: When is the solution unique?
Question 3: When do the numerical solutions {xk } converge
to that solution?
Question 4: How fast do the numerical solutions {xk }
converge to that solution?
Answer: Theory of the conditions for the convergence!
30 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Convergence theory of the fixed point method

Definition (Lipschitz condition)
A function g satisfies a Lipschitz condition on a set G if there
exists a Lipschitz constant L 0 such that
|g (x) g (y )| L |x y | , for all x, y G .
Furthermore, if g satisfies the Lipschitz condition with 0 L < 1,
then g is said to be a contraction on the set G .
Lemma (I)
If g satisfies a Lipschitz condition on G , then g is continuous on G .
Proof.
Inequality (1) directly gives the continuity based on its definition in
the calculus book.
31 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Definition (Cauchy sequence)
A sequence {xk }
k=1 is called a Cauchy sequence if, given any
> 0, there exists a N dependent on such that |xk xn | < for
every k and n greater than N.
Lemma (II)
Cauchy sequences in R and C must converge to a point in R or C.
Definition (Complete spaces)
Number systems in which Cauchy sequences must converge are
called complete spaces.
32 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Theorem (I: Contraction Mapping Theorem in one variable)
Suppose that g maps G into itself (i.e., if x G , then g (x) G )
and g satisfies a Lipschitz condition with with 0 L < 1(i.e., g is
a contraction on G ). Then there exists a unique fixed point z G
(i.e., z = g (z)), and the sequence {xk } determined by x0 G and
xk+1 = g (xk ) (k = 0, 1, 2, ) converges to z with error estimates
|xk z|
|xk z|
Lk
|x1 x0 | and
1L
L
|xk xk1 | .
1L
Remark
This theorem answers Questions 1, 2, and 3.
33 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Proof: We first consider the convergence of {xk }.
Since g maps G into itself and the sequence is determined by
x0 G and xk+1 = g (xk ) (k = 0, 1, 2, ), then
xk G (k = 0, 1, 2, ).
Since g satisfies a Lipschitz condition on G , then
|xk+1 xk | = |g (xk ) g (xk1 )| L |xk xk1 | .
By induction, we can get
|xk+1 xk | Lk |x1 x0 | .
34 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Similarly, for any n > k,
|xn xk |
nk
X
i=1
nk
X
|xk+i xk+i1 |
Li |xk xk1 |
i=1
L(1 Lnk )
|xk xk1 | .
1L
Since g is a contraction on G , then 0 L < 1, which implies

1 Lnk < 1.
35 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
So
L
|xk xk1 |
1L
L2
|xk1 xk2 |
1L

Lk
|x1 x0 | .
1L
|xn xk |
36 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
So
L
|xk xk1 |
1L
L2
|xk1 xk2 |
1L

Lk
|x1 x0 | .
1L
|xn xk |
{xk } is a Cauchy sequence!

Given any , there exists an N sufficiently large such that if
k, n > N, then |xn xk | < .
36 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Actually, we can obtain the N as follows.
(1 L)
|x1 x0 |
(1 L)
(1L)
)
ln(Lk ) ln( |x
) k ln(L) ln(
1 x0 |
|x1 x0 |
Lk
1L
|x1 x0 |
ln(
(1L)
|x1 x0 |
ln(L)
Lk
Here ln(L) < 0 if 0 < L < 1. As for L = 0, |xk xn | 0 <

for any k and n.
37 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Continued proof:
"
Then we can pick N =
ln(
(1L)
)
|x1 x0 |
ln(L)
#
+ 1, where [P] means the
nearest integer to P.
By Lemma (II), the sequence {xk } must converge.
38 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Now we prove the existence of the fixed point z.
Since the sequence {xk } must converge, we can assume
lim xk = z.
k
By Lemma (I), g is continuous since g satisfies a Lipschitz

condition.
Then taking the limit on both sides of xk+1 = g (xk ), we get
lim xk+1 = lim g (xk ).
And then z = g (z). Hence z is a fixed point and the sequence

{xk } converges to z.
39 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Now lets check the uniqueness of the fixed point z.
Assume that there are two different fixed points z1 and z2 .
We have z1 = g (z1 ) and z2 = g (z2 ).
Then
|z1 z2 | = |g (z1 ) g (z2 )| L |z1 z2 | .
Since g is a contraction on G , we have 0 L < 1. Recall that
z1 z2 6= 0. Then
|z1 z2 | < |z1 z2 | ,
which yields the contradiction. Therefore there exists a unique
fixed point z G .
40 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Recall
|xn xk |
L(1 Lnk )
|xk xk1 | ,
1L
and
|xn xk |
Lk
|x1 x0 | .
1L
Then let n , we get

|z xk |
L
|xk xk1 | ,
1L
and
|z xk |
Lk
|x1 x0 | .
1L
41 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
It may be difficult to find G such that g maps G to itself.
It may be difficult to show that g is a contraction on G .
Question: How to obtain the two conditions required by the
Contraction Mapping Theorem: g maps G to itself and g is a
contraction on G ?
42 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Theorem (II: Mean Value Theorem)

Suppose f C 1 [a, b], x [a, b], and y [a, b]. Then there exists
a [x, y ] [a, b] such that
f (y ) f (x) = f 0 ()(y x).
Here C 1 [a, b] denotes the space of the functions which are first
differentiable.
43 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Lemma (III)
Suppose that g is continuously differentiable and that |g 0 (x)| L
for x G . Then g satisfies a Lipschitz condition with Lipschitz
constant L for x G .
44 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Proof.
Consider x, y G .
By the Mean Value Theorem, we have
g (y ) g (x) = g 0 ()(y x).
Also, by |g 0 (x)| L for x G , we have |g 0 ()| L.
Combining these two conclusions, we obtain
|g (y ) g (x)| L |y x| .
This completes the proof.
45 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Another proof:
Consider x, y G .
Without loss of generality, assume that x < y . Then

Z y

0
g (s) ds
|g (x) g (y )| =
Z xy
0
g (s) ds
x
Z y
L ds
x
L |x y | .
46 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
The above lemma provides a technique to find the Lipschitz

constant L.
3
Example 6: Suppose g (x) = x6 +

constant L for g on G = [ 21 , 21 ].
2
x5
120 .
Find the Lipschitz
x
and
Solution: Note that g 0 (x) = x2 + 24
2
4
1
x
x
1
8 2 + 24 384 .
2

x4
We have |g 0 (x)| = x2 + 24
18 , thus L = 81 .
47 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Lemma (IV)
Suppose that g is continuously differentiable and that
|g 0 (x)| L < 1 for x G . Then g is a contraction on G .
48 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Lemma (V)
Let > 0 and G = [c , c + ]. Suppose that g is a contraction
on G with Lipschitz constant 0 L < 1 and
|g (c) c| (1 L). Then g maps G into itself.
49 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Lemma (V)
Let > 0 and G = [c , c + ]. Suppose that g is a contraction
on G with Lipschitz constant 0 L < 1 and
|g (c) c| (1 L). Then g maps G into itself.
Proof.
Consider x G . We have |x c| and
|g (x) c| = |g (x) g (c) + g (c) c|
|g (x) g (c)| + |g (c) c|
L |x c| + (1 L)
L + (1 L) = ,
thus g (x) [c , c + ] = G , which completes the proof.
49 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Theorem (III)
Assume that z is a solution of x = g (x), g 0 (x) is continuous in an
interval about z, and |g 0 (z)| < 1. Then g is a contraction in a
sufficiently small interval about z, and g maps this interval into
itself. Thus, provided x0 is picked sufficiently close to z, the
iterations will converge.
Remark
The above lemmas and theorem provide more details about
the conditions for the convergence than the Contraction
Mapping Theorem.
But they are sufficient conditions, not necessary conditions!
50 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Proof.
Since |g 0 (z)| < 1, there exists a L such that |g 0 (z)| L < 1.
Moreover, g 0 (x) is continuous in an interval about z, there
exists a sufficiently small interval I = [z , z + ] such that
maxxI |g 0 (x)| L < 1.
Then by Lemma (IV), g is a contraction on I .
And by Lemma (III) for x I ,
|g (x) z| = |g (x) g (z)| L |x z| ,
hence g maps I into I .
Therefore the Contraction Mapping Theorem can be used
with G = I to obtain the convergence of the iterations.
51 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Recall Example 2: g (x) = 4 + 31 sin 2x. What initial values
can give convergent numerical solutions of the fixed point
method?
Solution: First, g is continuous everywhere.
Second,
g 0 (x) =

2
2
cos 2x g 0 (x) < 1
3
3
for any x R.
Then by Lemma (IV), g is a contraction on G = R.
Obviously, g maps G = R into itself.
Thus, by the the Contraction Mapping Theorem, the fixed
point method is convergent for any initial value x0 G = R.
52 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Recall Example 3: g (x) = x2 + x1 . Can we prove that the fixed
point iteration converges for any initial value x0 G = [1, 2]?
Solution: First g is continuous except x = 0.
Second, g 0 (x) =
1
2
1
x2
and |g 0 (x)|
1
2
< 1 on G = [1, 2].
Then by Lemma (IV), g is a contraction on G = [1, 2] and

L = 12 .
Since G = [1, 2], pick up = 21 and c = 32 in Lemma (V).
Then

3
3
1
1

|g (c) c| = g ( ) =
= (1 L).
2
2
12
4
Then by Lemma (V), g maps G = [1, 2] into itself.
By the Contraction Mapping Theorem, the proof is completed.
53 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Recall Question 4: How fast do the numerical solutions {xk }

converge to the exact solution?
Definition (Convergence rate/order)
Let {xk } be a sequence convergent to x . If there exist constants
C and and an integer N such that |xk+1 x | C |xk x |
for all k N, then we say that the rate of convergence is of order
at least . The rate is said to be linear if = 1 with C < 1 and
quadratic if = 2.
54 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
From xk+1 = g (xk ), z = g (z) and the Lipschitz condition, we
get
|xk+1 z| = |g (xk ) g (z)| L |xk z|.
Hence the fixed point method has at least linear rate of
convergence if the conditions of the Contract Mapping Theory
are satisfied.
But is it possible to have a higher rate of convergence?
55 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Theorem (IV: Taylors Expansion)
Suppose that f C n+1 [a, b] and x0 [a, b]. Then for any
x [a, b], we have the following Taylors expansion of f (x) at x0 :
f (x) = Pn (x) + Rn (x),
where
Pn (x) =
n
X
1 (k)
f (x0 )(x x0 )k
k!
k=0
1
= f (x0 ) + f 0 (x0 )(x x0 ) + f 00 (x0 )(x x0 )2 +
2
1 (n)
+ f (x0 )(x x0 )n ,
n!
56 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Theorem (IV: Taylors Expansion(Continued))

Rn =
1
f (n+1) ()(x x0 )n+1
(n + 1)!
for some [x0 , x] (Lagrange form of the remainder) ,
or
Rn
Z
1 x (n+1)
=
f
(s)(x s)n ds
n! x0
for some [x0 , x] (Integral form of the remainder) .
57 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Theorem (V)
Assume that the iterations xk+1 = g (xk ) converge to a fixed point
z and g C q (G ) where G contains z. Furthermore, assume that q
is the first positive integer for which g (q) (z) 6= 0 and if q = 1 then
g 0 (z) < 1. Then the sequence {xk } converges to z with order q.
Remark
Usually g 0 (z) 6= 0, hence q = 1.
Its not easy to find g (q) (z) (q = 1, 2, 3, ) since the
solution z is unknown.
58 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Proof:
Using xk+1 = g (xk ) and the Taylors expansion of g (xk ) at z,
for some k between xk and z, we have
|xk+1 z| = |g (xk ) z|

= g (z) + g 0 (z)(xk z) +
1
g (q1) (z)(xk z)(q1)
+
(q 1)!

1
+ g (q) (k )(xk z)q z
q!

1 (q)
= g (k ) |xk z|q .
q!
59 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Continued proof:
Therefore

1 (q)

|xk+1 z|
sup g (x) |xk z|q
xG q!
= C |xk z|q ,

which implies that the sequence {xk } converges to z with

order q.
For linear convergence, we need C < 1. And the assumption
if q = 1 then g 0 (z) < 1 guarantees C < 1 when q = 1.
60 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
61 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Basic idea and derivation of the Newtons method
Still consider f (x) = 0.

f (x) could be complicated and nonlinear.
Any idea to approximate f (x) by using a simpler function?
Linearization by using Taylors expansion!
62 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Taylors expansion:
f (x) = f (x0 ) + f 0 (x0 )(x x0 ) + 21 f 00 ()(x x0 )2
Then the linearization is f (x) p(x) = f (x0 ) + f 0 (x0 )(x x0 ).
Replace f (x) by p(x) = f (x0 ) + f 0 (x0 )(x x0 ) in f (x) = 0.
Then f (x0 ) + f 0 (x0 )(x x0 ) = 0.
Hence x = x0
f (x0 )
f 0 (x0 ) .
But the error is big since 21 f 00 ()(x x0 )2 could be big.

Is there anyway to reduce the error?
Yes, iteration xk+1 = xk
f (xk )
f 0 (xk ) !
63 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Another explanation:
Use the tangent line of f (x) at x0 to approximate f (x):
y f (x0 ) = f 0 (x0 )(x x0 ) y = f (x0 ) + f 0 (x0 )(x x0 ).
Then use the root of the tangent line to approximate the root
0)
of f (x). We solve y = 0 to obtain: x1 = x0 ff 0(x
(x0 ) .
Repeat the same procedure for x1 , x2 , x3 , . That is, after
we obtain xk , we use the tangent line of f (x) at xk to
approximate the curve of f (x). Then use the root of the
tangent line to approximate the that of f (x).
Hence we get xk+1 = xk
f (xk )
f 0 (xk ) !
64 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
A geometric explanation of the Newtons method: see Figure
3.4 on page 51 of the textbook.
This is a special fixed-point method with g (x) = x
f (x)
f 0 (x) .
Its dangerous when f 0 (xk ) is close to 0.
65 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics

Remark
The Newtons method is quadratically convergent: Suppose z
is the fixed point, then g (z) = z and f (z) = 0. If f 0 (z) 6= 0,
then
f (x)f 00 (x)
[f 0 (x)]2 f (x)f 00 (x)
=
[f 0 (x)]2
[f 0 (x)]2
00
f (z)f (z)
g 0 (z) =
= 0.
[f 0 (z)]2
g 0 (x) = 1
But g 00 (z) may not be zero. Thus we can obtain the quadratic
convergence rate based on the theorem (V).
The quadratic convergence rate also matches the second order
remainder 12 f 00 ()(x x0 )2 in the above Talyors expansion.
66 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Stop criteria
Let , f and f 0 be the tolerance
|xk+1 xk | < ;
|f (xk )| < f ;
|f 0 (xk )| < f 0 ;
Any combination.
67 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Algorithm of the Newtons method

1
(1) Pick up the initial point x0 .
(2) Specify the tolerances , f , f 0 , and the maximum number
M of iteration steps.
f (x0 )
f 0 (x0 ) .
First step: k = 0, x1 = x0

(1) k k + 1.
(2) xk+1 = xk ff0(x(xkk)) .
(3) If |f (xk+1 )| < f or |f 0 (xk+1 )| < f 0 , then stop the loop.
numerical solution.
68 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Matlab for the Newtons method
Example 1: f (x) = e x + x. Choose x0 = 0, = 0.01,

f = 108 , f = 108 , and M = 100.
69 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Numerical results:
k
0
1
2
3
xk
0
0.5
0.566311
0.567143
Verification: f (0.567143) 4.551138 107 .
70 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Comparison among the Newtons method, the bisection method

and the fixed point method:
Verification for the Newtons method after 3 iterations:
f (0.567143) 4.551138 107 .
Verification for the bisection method after 6 iterations:
f (0.5703125) 4.963760 103 .
Verification for the fixed point method after 10 iterations:
f (0.564879) 3.549922 103 .
71 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
72 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Basic idea and derivation of the secant method

The Newtons method needs to compute f 0 (xk ). But in
realistic circumstances, it may not be possible to compute
f 0 (x) explicitly.
This is actually the reason why numerical differentiations have
been developed.
One traditional numerical differentiation is
f (x )f (x
)
f 0 (xk ) xkk xk1k1 , which is based on the definition of the
derivative: f 0 (x) = lim
y x
f (x)f (y )
.
xy
Then we replace the f 0 (xk ) in the Newtons iteration by the

above numerical differentiation to obtain
f (x )(x xk1 )
xk+1 = xk f (xkk )fk (xk1
).
73 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
Compared with the Newtons method, the secant method uses
the line determined by xk and xk1 , not the tangent line of
f (x) at xk , to approximate the curve of f (x). Then we use
the root of the line determined by xk and xk1 to approximate
the that of f (x).
A geometric explanation of the secant method.
74 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Remark
The secant method is not a fixed-point method. It involves
not only xk+1 and xk , but also xk1 .
For the secant method, we need two initial values x0 and x1 .
But we need only one function evaluation f (xk ) at each
iteration step since f (xk1 ) was computed in the previous
iteration step.
75 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Stop criteria
The implementation issues of the secant method is similar to those

of the Newtons method, but we dont need the tolerance for the
f 0 (xk ) any more. Let and f be the tolerance
|xk+1 xk | < ;
|f (xk )| < f ;
Any combination.
76 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Algorithm of the secant method

1
(1) Pick up the initial points x0 and x1 .
(2) Set the iteration counter k = 0.
(3) Specify the tolerances , f , and the maximum number M
of iteration steps.

(1) k k + 1.
k xk1 )
(2) xk+1 = xk ff(x(xkk)(x
)f (xk1 ) .
(3) If |f (xk+1 )| < f , then stop the loop.
numerical solution.
77 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Matlab for the secant method
Example 1: f (x) = e x + x. Choose x0 = 0, x1 = 1, = 0.01,

f = 108 , and M = 100.
78 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Practice in Matlab for the secant method
Numerical results:
k
0
1
2
3
4
5
xk
0
1
0.367879
0.498592
0.564573
0.567111
Verification: f (0.567111) 5.060399 105 .

Similar performance to the Newtons method.
79 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Convergence theory of the secant method
A bit on convergence: One can show that
1
= (1 + 5) 1.62.
2
This is called super linear convergence (1 < < 2).
|xk+1 x | C |xk x | ,
See pages 60-62 of the book by A. S. Ackleh et al. (Independent

study problem)
80 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Outline
Introduction
Bisection method
Newtons method
Secant method
Other topics
81 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Topics: Independent study problems
How to deal with multiple zeros?

Different modified Newtons methods: the simplified Newtons
method, the Newtons method with a downhill parameter, the
interval Newtons method, and so on.
Solving nonlinear equations with Newtons by C. T. Kelly,
Fundamentals of Algorithms, 1. Society for Industrial and
Applied Mathematics (SIAM), Philadelphia, PA, 2003.
82 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Topics: Independent study problems
The M
ullers method: Compared with the secant method , the
M
ullers method uses a parabola, which is determined by xk ,
xk1 and , xk2 , to approximate the curve of f (x). Then we
use the root of the parabola to approximate the that of f (x).
Aitken acceleration : Combine the formulas of two or more
iteration steps to obtain an approximation with higher rate of
convergence.
Many other methods......
83 / 85
Introduction
Bisection method
Newtons method
Secant method
Other topics
Reasons to begin with f (x) = 0
This problem occurs frequently.

This problem provides us a quick but complete look at
different numerical methods for a problem.
Solving this problem illustrates the iterative method, which is
a common numerical technique.
Moreover, it illustrates that when many methods are available,
selection of the most suitable method depends on the specific
problem, the goal, and the computation equipments available.
84 / 85

Numerical Analysis Chapter 2

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Numerical Analysis Chapter 2

Diunggah oleh

Hak Cipta:

Format Tersedia

Introduction

The fixed point method

Numerical Analysis and Computation

The fixed point method

The fixed point method

The fixed point method

The fixed point method

The fixed point method

The fixed point method

Reasons to begin with f (x) = 0

This problem occurs frequently.

The fixed point method

Existence of the solution

Theorem (Intermediate Value Theorem)

The fixed point method

Existence of the solution

The fixed point method

Uniqueness of the solution

The fixed point method

The fixed point method

The fixed point method

Basic idea of the bisection method

The fixed point method

Basic idea of the bisection method

Once we identify the half interval which contains z, we define

The fixed point method

Basic idea of the bisection method

Once we identify the half interval which contains z, we define

The fixed point method

Basic idea of the bisection method

Once we identify the half interval which contains z, we define

The fixed point method

Algorithm of the bisection method

(1) Define a0 = a and b0 = b.

, repeat the following procedure in a loop:

Output the final xk from the above loop.

The fixed point method

Matlab for the bisection method

Several implementation issues:

The fixed point method

Interval [ak , bk ] is small enough;

Max number of iterations is reached;

Any combination of the previous ones.

The fixed point method

Matlab for the bisection method

Example 1: f (x) = e x + x, f (0) = 1, and f (1) = 0.632.

The fixed point method

Matlab for the bisection method

Verification: f (0.5703125) 4.963760 103 . A smaller

The fixed point method

Convergence theory of the bisection method

we finish the proof.

The fixed point method

Convergence theory of the bisection method

Hence 14 iteration steps are required.

The fixed point method

Convergence theory of the bisection method

The fixed point method

The fixed point method

The fixed point method

Basic idea of the fixed point method

The fixed point method

Let and f be the tolerance