Phys7411: Computational

Physics (with Python)

Prof: Mark Wilde
Textbook: Computational Physics (Mark Newman)
Typesetting: Luke Bouma

2015/07/21 – 2015/08/07

1
Contents

1 Lectures 1-5 5
1.1 Lecture 1: First Programs, Rydberg Lines . . . . . . . . . . . 5
1.1.1 On to Ch 2 . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Lecture 2: Basics, contd . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 Height of ball, dropped from tower height h . . . . . . 8
1.2.2 Packages . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.3 Example: Converting Polar to Cartesian . . . . . . . . 8
1.2.4 Built-in Functions: if, elif, else, while, break . . . . . . 8
1.3 Lecture 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.1 Multidimensional Arrays . . . . . . . . . . . . . . . . . 11
1.3.2 Reading Arrays from Files . . . . . . . . . . . . . . . . 11
1.3.3 Array Arithmetic . . . . . . . . . . . . . . . . . . . . . 11
1.3.4 Slicing Elements from Lists or Arrays . . . . . . . . . 13
1.3.5 For loops . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4 Lecture 4: for loops, range, plotting . . . . . . . . . . . . . . 14
1.4.1 Using for loops to compute sums . . . . . . . . . . . . 14
1.4.2 User-defined Functions . . . . . . . . . . . . . . . . . . 15
1.4.3 Good programming style . . . . . . . . . . . . . . . . 16
1.4.4 Graphics: Simple Plots (Ch.3) . . . . . . . . . . . . . 17
1.4.5 Importing data from file and plotting . . . . . . . . . 18
1.4.6 Changing features of graphs . . . . . . . . . . . . . . . 18
1.4.7 Scatter Plots . . . . . . . . . . . . . . . . . . . . . . . 19
1.4.8 Density plot . . . . . . . . . . . . . . . . . . . . . . . . 20
1.5 Lecture 5: wave interference, Mandelbrot set . . . . . . . . . 20
1.5.1 Plotting the Mandelbrot set . . . . . . . . . . . . . . . 22
1.5.2 Accuracy and Speed (Ch.4) . . . . . . . . . . . . . . . 22
1.5.3 Numerical error . . . . . . . . . . . . . . . . . . . . . . 23
1.5.4 Program Speed . . . . . . . . . . . . . . . . . . . . . . 25
1.5.5 Matrix multiplication . . . . . . . . . . . . . . . . . . 25

2
2 Lectures 6-10 27
2.1 Lecture 6: 3D plotting with visual package, integral approxi-
mation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2 Chapter 5: Integration (lec 6) . . . . . . . . . . . . . . . . . . 28
2.2.1 Trapezoidal rule . . . . . . . . . . . . . . . . . . . . . 28
2.2.2 Simpson’s Rule . . . . . . . . . . . . . . . . . . . . . . 30
2.2.3 Quantifying error bounds . . . . . . . . . . . . . . . . 31
2.3 Lecture 7: Integration techniques . . . . . . . . . . . . . . . . 32
2.3.1 Choosing the Number of Steps for an Integral . . . . . 34
2.3.2 Romberg Integration . . . . . . . . . . . . . . . . . . . 36
2.4 Lecture 8: Higher-order integration methods, and Gaussian
quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.1 Gauss Quadratures . . . . . . . . . . . . . . . . . . . . 38
2.5 Lecture 9: Choosing integration methods; integrals for infi-
nite ranges; derivatives . . . . . . . . . . . . . . . . . . . . . . 43
2.5.1 Computing integrals over infinite ranges . . . . . . . . 43
2.5.2 Multiple Integrals . . . . . . . . . . . . . . . . . . . . 45
2.5.3 Derivatives (finite difference and more) . . . . . . . . 46
2.6 Lecture 10: Higher order derivatives, interpolations, Gaussian
elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.6.1 Interpolation . . . . . . . . . . . . . . . . . . . . . . . 51
2.6.2 Chapter 6! Solving linear equations . . . . . . . . . . 52

3 Lectures 11-15 54
3.1 Lecture 11: LU decomposition . . . . . . . . . . . . . . . . . 55
3.1.1 Calculating matrix inverses . . . . . . . . . . . . . . . 57
3.1.2 Eigenvalues and eigenvectors: QR algorithm . . . . . . 59
3.1.3 Description of QR algorithm . . . . . . . . . . . . . . 61
3.2 Lecture 12: Solving nonlinear equations . . . . . . . . . . . . 62
3.2.1 Rate of convergence for relaxation method . . . . . . . 64
3.2.2 Relaxation method for 2 or more variables . . . . . . . 65
3.2.3 Bisection method . . . . . . . . . . . . . . . . . . . . . 66
3.2.4 Newton’s method . . . . . . . . . . . . . . . . . . . . . 67
3.3 Lecture 13: Secant method; generalized Newton’s; semidefi-
nite programming . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.1 Newton’s method for multiple variables . . . . . . . . 69
3.3.2 Semidefinite programming . . . . . . . . . . . . . . . . 74
3.4 Lecture 14: more semidefinite programming . . . . . . . . . . 75
3.5 Lecture 15: Fourier series and transforms . . . . . . . . . . . 76
3.5.1 Discrete Fourier transforms . . . . . . . . . . . . . . . 77

3
 3.5.2 2D Fourier transform . . . . . . . . . . . . . . . . . . . 80

4 Lectures 16-20 83
4.1 Lecture 16: Fast Fourier transform, ordinary diffeqs, Euler
method, Runge-Kutta . . . . . . . . . . . . . . . . . . . . . . 83
4.1.1 Ch. 8: Solving Ordinary Differential Equations . . . . 86
4.2 Lecture 17: Simultaneous ordinary diffeqs, adaptive step size
Runge-Kutta, leap frog method. . . . . . . . . . . . . . . . . . 93
4.2.1 Second order and higher diffeqs . . . . . . . . . . . . . 94
4.2.2 Other variations: leap-frog method . . . . . . . . . . . 100
4.3 Lecture 18: Leap frog & energy conservation, Verlet method,
modified midpoint method, Bulirsch-Stoer technique . . . . . 101
4.3.1 Verlet method . . . . . . . . . . . . . . . . . . . . . . 103
4.3.2 Modified midpoint method . . . . . . . . . . . . . . . 105
4.3.3 Bulirsch-Stoer Method . . . . . . . . . . . . . . . . . . 107
4.4 Lecture 19: Boundary value problems, shooting method, par-
tial diffeqs, finite differences . . . . . . . . . . . . . . . . . . . 109
4.4.1 Shooting method . . . . . . . . . . . . . . . . . . . . 109
4.4.2 Chapter 9: Partial differential equations. . . . . . . . . 112
4.4.3 Method of finite differences . . . . . . . . . . . . . . . 113
4.5 Lecture 20: Gauss-Seidel, initial value problems, forward time
centered space method, numerical stability analysis. . . . . . 119
4.5.1 Gauss-Seidel method (GS) . . . . . . . . . . . . . . . . 119
4.5.2 Initial value problems . . . . . . . . . . . . . . . . . . 120
4.5.3 Numerical stability analysis . . . . . . . . . . . . . . . 123

5 Assessment and next steps 129

4
Chapter 1

Lectures 1-5

This project is an exercise to: (1) learn how to approach and solve physics
problems with computers, (2) develop fluency with the BaKoMA TeX editor,
and (3) develop fluency with Python. We follow material available from
Mark Wilde’s website, available here.

1.1 Lecture 1: First Programs, Rydberg Lines

Student Introduction : I’m Luke Bouma. I want to understand physical
reality. Shy of analytic solutions, this means I need to approximate solutions
to interesting problems. This course is essential given this understanding;
modern science uses computers. If I’m going to be a literate scientist, then
I’ll need to at least know about computational methods.
Begin notes: Computation is an indispensable tool in modern physics.
Nearly every problem is solved with the aid of computers. Such problems
include:

• Numerical integration

• Inverting large matrices

• Solving nonlinear differential equations

Standard course timeline is 1 month to learn Python, 1 month on numerical

techniques, and 1 month on applications to physical problems. I’ve had
a good exposure to numerical techniques, but the main aims should be
producing good homework sets and complete lecture notes.

5
 Simple example of Python code: Rydberg formula for wavelengths of
emission lines of hydrogen:
 
1 1 1
=R − (1.1)
λ m 2 n2

where R, the Rydberg constant, is 1.097×10−2 nm−1 and n, m are positive in-
tegers. For fixed m, and varying n > m we get a series. A Python program
that generates the first five lines of the first three series would look like the
following:
R = 1 . 0 9 7 e−2
for m in [ 1 , 2 , 3 ] :
p r i n t ( ” S e r i e s f o r m=” , m)
for k in [ 1 , 2 , 3 , 4 , 5 ] :
n = m + k
invlambda = R∗ ( 1 /m∗∗2 − 1/n ∗ ∗ 2 )
p r i n t ( ” ” , 1 / invlambda , ”nm” )

A couple things to note about the program are (1) indention shows pro-
gram flow, (2) for loop and list syntax, (3) assignment of variables, (4) math
calculations, (5) output syntax.

1.1.1 On to Ch 2
Go get the following program to run on your computer:
x=1
print (x)

I chose the PyCharm IDE since it’s cross-platform (these notes are being
written on Windows 8, but I want to be able to develop on *nix systems as
well). We’re going to be using Python3.4.3 as our interpreter.
Some types of variables that Python has (n.b., its data-types are not
explicit, which is nuts after living in C++ land) include integers, floats,
complex numbers, etc. It’s important to have these different types both
for human understanding, but also to deal with memory issues; integers are
cheaper than floats which are cheaper than complex. This contributes to
how long it takes to do calculations as well. There’s also the question of
accuracy: computers have a finite amount of it, so it’s best to use integer
types when the quantity is a genuine integer. In Python,
x = 1 # i n t e g e r type
x = 1.5 # f l o a t type
x = 1.0 # f l o a t type
x = float (1.0) # f l o a t type

6
x = 1.5 + 0 j # complex type
x = complex ( 1 . 5 )# complex type
x = ” string ” # s t r i n g type

and then there are also string types.

Before going on to lecture 2, go to test2.py, and try running some com-
mands!

1.2 Lecture 2: Basics, contd

The basic arithmetic operations are what you would expect for addition,
subtraction, multiplication and division. For exponentiation, use x ∗ ∗y.
Take care to notice that the type of a result is determined by the type of
inputs: the output will be the more general of the types. Also, division
never gives integer results in Python 3. Some other operations:
x // y #i n t e g e r p a r t o f x , d i v i d e d by y
x % y #modulo− r e m a i n d e r o f x/y
3∗ x ∗∗2 # 3x ˆ 2 ; can combine a s e x p e c t e d

What else? well the statement

x = x+1

assigns x to be one more than it was before. Shortcuts for this and similar
assignments are:
x += 1 #adds 1 t o x
x −= 4
x ∗= −2.6
x /= 5∗ y # x = x /5 y
x //= 3 . 4 # d i v i d e x by 3 . 4 , rnd dwn , a s s i g n t o x

Another feature of Python is assigning multiple variables in a single line,

like:
x , y = 1 ,2.5 # same a s two s e p a r a t e l i n e s
x , y = 2∗ z +1 , ( x+y ) /3

for the latter line, all the RHS is evaluated before the left hand side. This
means that the statement
x,y = y,x

is simultaneous, so counts as a variable swap.

7
1.2.1 Height of ball, dropped from tower height h
The ball’s height as a function of time is y(t) = h − 12 gt2 . Bring up
dropped.py, and run it. The program should work. Note it’s much better
practice to assign g = 9.81 and then write s = g ∗ t ∗ ∗2/2 than what is
actually in the file online.

1.2.2 Packages
Python comes with packages. One important one is math. The call function
for say, the log function from math is written
from math import l o g

which lets you compute the natural logarithm. Other common math func-
tions include log, log10, exp, sin, cos, tan, asin, acos, atan, sinh, cosh, tanh,
sqrt. For sqrt, you could use x ∗ ∗0.5, but the function is quicker and more
accurate. To import multiple functions, use
from math import l o g , exp , s i n , . . . #OR u s e below
from math import ∗

but it’s usually better not to, so that you stay aware of everything that you’re
using. Some packages have modules, which are like smaller subpackages. For
instance, numpy has as linear algebra module, which you can import as
from numpy . l i n a l g import i n v

1.2.3 Example: Converting Polar to Cartesian

Write a program to convert polar to Cartesian coordinates, where the user
enters (r, θ) with θ in degrees, and you output (x, y). Bring up polar.py
and compare your solution.

1.2.4 Built-in Functions: if, elif, else, while, break

Built-in functions are always available, so there’s no need to import them.
Some examples of built-in functions are float, int, complex, abs, import,
print. There are also conditional statements, like if. When you’re working
with those (basically, when you’re controlling program flow) it’s critical to
indent your code appropriately. The syntax for these is
i f ( x>10 o r ( x<1 and x>−5) o r x==2 o r x!=3
statement

for instance,

8
x = i n t ( i n p u t ( ” Enter a whole number no g r e a t e r than t e n : ” ) )
i f x > 10:
p r i n t ( ”Your number i s g r e a t e r than t e n . ” )
e l i f x >9:
p r i n t ( ”Your number i s OK, but you ’ r e c u t t i n g i t c l o s e . ” )
else :
p r i n t ( ”Your number i s f i n e . Move a l o n g . ” )

While statements keep loops running while a given condition is true.

Check out 6-while.py for an example of this. You can get out of while loops
with break statements, which you can trigger with conditional statements
as in 7-while-break.py. This is a program that illustrates an if statement
nested inside of a while loop.
There are two more programs to study: 8-even-odd.py and 9-fibonacci.py.
The first of these has a while loop that ends once the remainder of (m+n)/2
is zero, i.e., when m is even and n is odd, or vice versa. The second is a
while loop that prints the first Fibonacci sequence, Fn+1 = Fn + Fn−1 until
the terms go above 1000.
f1 , f 2 = 0 ,1
w h i l e f 2 <1000:
print ( f2 )
f 1 , f 2 = f 2 , f 1+f 2

Note that the nice thing about this code is the simultaneous assignment in
the fourth line. What might take extra lines or a placeholder-variable in
other languages is single-line awesomeness is Python. Nice.

1.3 Lecture 3
How do we represent sequences of numbers? Python says: lists, or arrays.

Lists consist of a list of quantities of a given type. The elements don’t

have to be of the same type, although they usually are. Write them as
[3 ,0 ,0 , −7 ,24]

or
r = [ 1 , 2 . 5 , 3+4.6 j ]
print ( r )

or you can also do

x , y , z = 1 , 3.2 , 5 j
r = [x,y,z]

9
Similarly, it is possible to calculate elements of a list from mathematical
expressions. Then the syntax to access elements of the list is r[0], r[1], etc.
Note that this is zero-based counting.
Bringing out 1-list-sqrt.py, we see this in action to calculate the
length of a 3d vector. Some built-in functions to operate on lists are: sum,
max, min, len. Bringing up 2-list-avg.py, we can see the sum and len.
Sum adds up all the components of a list, and len returns the number of
elements in the list.
What if we want to apply a math function to every element of a list?
For instance, we have a list r and we want to take the natural log of every
element. We do this with the “meta function”, map. We would write
map( l o g , r )

which creates an objected called “iterator” in memory, which can be con-

verted to another list. To see this in action, bring up 3-log-map.py. The
command winds up being of the form
l o g r = l i s t (map( l o g , r ) )

Similarly, we use commands of the form r.append(1) to append 1 as the

final element of the list r. If we want to remove the third element of the
list, we would write r.pop(2). Note though that this is a slow operation, and
shouldn’t be used much.
We can also create empty lists, and then add to them later:
r = []
r . append ( 1 . 0 )
r . append ( −2.2) . . .

Arrays are an alternative to lists. The differences between them are:

1. The number of elements in an array is fixed. Can’t add/remove.

2. Elements in array must all be of same type.

Arrays also have some advantages over lists, including

• Arrays can be 2-dimensional

• Arrays behave like vectors/matrices, for addition/subtraction

• Arrays are way faster than lists

10
Given all these benefits, in physics we’ll normally use fixed arrays of ele-
ments, all of the same type. This is also helpful because array functions are
part of the package numpy.
Bring up 4-create-array.py. This file requires the numpy library. If
you haven’t already installed it, do so. StackExchange is your friend (on
*nix this is easy. On Windows, this is less easy). You see that
zeros (10 , int ) #c r e a t e s a r r a y l e n g t h 1 0 , type i n t
z e r o s ( 1 0 , complex ) #same , with type complex

If you don’t need zeros to begin with, use

a = empty ( 5 , f l o a t )

which initializes whatever there is to memory.

To convert a list into an array, write
r = [ 1 . 0 , 1 . 5 , −2.1]
a = array ( r , f l o a t ) #from l i s t t o a r r a y
r = l i s t (a) #from a r r a y t o l i s t

1.3.1 Multidimensional Arrays

You can create a 2d array from a list of lists. The syntax is similar to
Mathematica, if you’ve seen that before:
a = array ( [ [ 1 , 2 , 3 ] , [ 4 , 5 , 6 ] ] , int )

and then accessing elements is just writing a[i, j] for the ith row and the
j th column. This works the same for assignments. If you want a 2 × 3 × 4
tensor, as an array in Python you create it as
m = array ( [ 2 , 3 , 4 ] , f l o a t )

1.3.2 Reading Arrays from Files

A super-common thing that’ll happen in Python is that you’ll have some
kind of text file with data in it that you’ll need to pipe into Python. This
is done with a function called loadtxt. This function can load both 1d and
2d arrays of numbers into arrays. The text can have different delimiters as
well; documentation for loadtxt gives the different options.

1.3.3 Array Arithmetic

Look at 6-arith-array.py. You’ll see you can do standard matrix arith-
metic with arrays. The syntax for manipulating rows/columns of arrays is

11
analogous to Matlab/Octave, e.g., a = v[: 1] accesses the second column of
v. N.b., Python is zero-based counting.
Multiplying two vectors doesn’t give an inner/outer product, but instead
gives the element-wise product.
To take dot products, bring up 7-mult-array.py. You’ll see that
Numpy has a dot(a,b) function, that does exactly what you’d expect for
1d vectors. You can also use it to multiply a vector with a matrix, and
vice-versa so that dot(M, v) is the right-product, and dot(v, M ) is the left
product. Notice that the transpose operation is implicit!
We can apply all the Numpy functions we’ve seen, max, min, sqrt as well
as map to arrays as well. E.g., if we wanted a square root of the elements
in an array, we would write
b = a r r a y (map( s q r t , a ) , f l o a t )

Note the program in 9-array-size-shape.py is wrong! The list conver-

sion seems to not be able to convert mapped objects (in memory) to lists as
desired. A program that does the same thing is:
import numpy a s np
a = np . a r r a y ( [ [ 1 , 2 , 3 ] , [ 4 , 5 , 6 ] ] , f l o a t )
p r i n t ( np . s q r t ( a ) )

Moving on, looking at 10-means-arith-geo.py we can get an idea for

why processing data in Python can be a good idea. In this code, we’re
calculating a geometric mean,
" n
#1/n
Y
ˉ=
x xi (1.2)
i=1
ˉ = exp(log(ˉ
x x)) (1.3)
n
!!
1 Y
= exp log xi (1.4)
n
i=1
n
!
1X
= exp log(xi ) (1.5)
n
i=1

(I’ll figure out BaKoMa aligning later). There’s another error in this code
(for Python3.4.3), in the line that should be
l o g s = a r r a y ( l i s t (map( l o g , v a l u e s ) ) , f l o a t )

or else Numpy’s array function doesn’t know how to convert from memory
(i.e., it seems map points to a location in memory) to arrays.

12
 A much less sloppy way to do this is in 11-alt-geo-mean.py, which
imports the log function from Numpy, and uses that on the values loaded
in from values.txt.
A word of caution! Consider the program
from numpy import a r r a y
a = array ( [ 1 , 1 ] , int )
b = a
a[0] = 2
print (a , b)

the output for this program will be [2, 1], [2, 1]! (not, as you might expect
for an ordinary assignment, [2,1],[1,1]). What’s happening is that b = a just
sets b to be another name for a. It doesn’t copy the contents of a into b – so
array assignments in Numpy are like pointer assignments in C. To copy, do
from numpy import copy
b = copy ( a )

1.3.4 Slicing Elements from Lists or Arrays

Bring up 12-slicing.py. You’ll see similar slicing as from Matlab/Octave:
r = [1 ,3 ,5 ,7 ,9 ,11 ,13]
s = r [:] #a s s i g n s a l l e l e m e n t s o f l i s t r
s = r [2:] #a s s i g n s from 3 rd e l e m e n t on
s = r [2:4] #a s s i g n s e l e m e n t s 3−5

which also works the same way for arrays.

1.3.5 For loops

The most common kind of loop we’ll use. Bring up 13-simple-for.py. The
program first makes a list. For each value in the list, it executes, and then
goes to the next value in the list. Indentation is important. We can use
break or continue commands to either break out, or continue in a for loop
until the next iteration.
For longer lists with obvious values to loop through, use the range func-
tion, which yields
r = range (5)
for n in r :
print (n) #p r i n t s 0 , 1 , 2 , 3 , 4

13
1.4 Lecture 4: for loops, range, plotting
For loops and the range function were introduced at the end of lecture 3.
The range function has some extra options:
range (5) #[ 0 , 1 , 2 , 3 , 4 ]
range ( 2 , 8 ) #[ 2 , 3 , 4 , 5 , 6 , 7 ]
range ( 2 , 2 0 , 3 ) #[ 2 , 5 , 8 , 1 1 , 1 4 , 1 7 ]
range (20 ,2 , −3) #[ 2 0 , 1 7 , 1 4 , 1 1 , 8 , 5 ]

For instance, a program that prints the first ten powers of two would be:
f o r i in range (1 ,11)
p r i n t (2∗∗ i )

where you should note the upper limit of the range is given as 11. Note that
all arguments passed to range should be integers.
A function similar to range from the Numpy package is called arange
which does the same thing, but returns arrays instead of lists. It also ac-
cepts floats as input, so you could write arange(start, stop, step) with float
arguments. In the spirit of being consistent with Matlab function names,
there’s a function called linspace from Numpy that does the same thing.

1.4.1 Using for loops to compute sums

Suppose we want
100
X 1
s= ,
k
k=1

and bring up 3-sum-for.py. This program compute the sum, with a for
loop. We could do the same thing by loading data from a file, as in
4-sum-file.py. Note that if we load the data into arrays, using the Numpy
command sum will usually be more efficient (as well as more compact to
read) than using for loops. Now we return to the hydrogen emission lines pro-
gram, and write a simpler version using range (see 5-emission-lines.py):
R = 1 . 0 9 7 e−2
f o r m in range ( 1 , 4 ) :
p r i n t ( ” S e r i e s f o r m=” ,m)
f o r n i n r a n g e (m+1 ,m+6) :
invlambda = R ∗ ( 1 /m∗∗2 − 1/n ∗ ∗ 2 )
p r i n t ( ”n =” , n , ” : ” , 1/ invlambda , ”nm” )

14
1.4.2 User-defined Functions
We can write and define our own functions. What kind of programming
language would this be if we couldn’t? This is obviously important, to avoid
writing the same code over and over again. For instance, say we want a
function to return the factorial of an integer that we pass to it,
n
Y
n! = k.
k=1

Bring up 6-factorial-function.py to see how this is done. The code

for defining the function is
def f a c t o r i a l (n) :
f = 1.
f o r j i n r a n g e ( 1 , n+1) :
f ∗= j
return f

and then calls to the function work as you would expect. Note that variables
created inside the function are local to the function; if you try to print them
outside the function, you’ll get an error since they won’t be in memory.
User-defined functions can take more than one argument. Look at cylin-
drical.py to see this in action. The arguments that a function takes can be
different types (int, float, complex, lists, arrays) and can also return any
type. For instance,
def cartesian ( r , theta ) :
x = r ∗ cos ( theta )
y = r ∗ sin ( theta )
return [ x , y ]

User-defined functions can also return multiple values. This uses the
Python single-line assignment functionality of
x,y = a,b

so if we wrote
def f ( z ) :
...
return a , b
x , y = f (3)

that would work in the same way.

Additionally, user-defined functions need not return any value. The func-
tion will end by respecting indeentation. Why would you want to do this?

15
For instance, this could be useful if you were printing elements of a vec-
tor/array repeatedly, and wanted a generic print function.
You can define a user-defined function anywhere in the program, so long
as the definition happens before it is used. Moreover, you can use user-
defined functions inside of other user-defined functions, and can even use
them with the map command!
To see this in action, bring up 8-user-map.py. In this program, you can
see that once you’ve defined a function, you can map it onto say, a list, and
then use that to define a new list.
You can go one better than this as well. Say you have a lot of functions to
define. Write them out in a file called mydefs.py, and then at the beginning
of your program write
from mydefs import myfunc

and the function myf unc will become available for use. Often, when import-
ing functions from a library this is what’s happening (although sometimes
you’re actually calling C-code, since it’s faster).
Another example of a user-defined function is in 9-prime-factors.py,
where the algorithm used returns prime factors of any integer input.

1.4.3 Good programming style

1. Use comments

2. Use meaningful variable names

3. Use the right type of variables

4. Import functions first

5. Give physical constants names

6. Employ user-defined functions when necessary

7. Print out partial results throughout the code

8. Lay out programs clearly

E.g., split long program lines into multiple ones using backslash \,
which tells Python the next line is part of the current one

9. Don’t make programs unnecessarily complicated

16
1.4.4 Graphics: Simple Plots (Ch.3)
Our first focus will be on making simple graphs. We will use the pylab
package which is part of a larger package called matplotlib. This student
is dubious that this’ll hold a candle to Gnuplot, but for the sake of single-
environment integration, let’s give it a shot.
Installing pylab requires previous installation of scipy. Once you’ve
worked out how to get it up and running, 10-simple-plot.py should return
a figure like:

which doesn’t seem as bad as I feared. Note for the commands that
plot(f ) plots it, but show() is required to get an output the user can see.
Why would there be two functions, one to plot, and one to show? Be-
cause you might want to plot multiple curve in the same figure, which you
could do by calling plot several times, then show all at once. Bring up
12-plot-sin.py, and modify it to plot both a sine and cosine. You should
get something like this:

17
 which, again, doesn’t look terrible. The program uses linspace to get
sample points for the x axis, and then makes a curve that looks like sin,
although the points are connected by straight line segments.

1.4.5 Importing data from file and plotting

Look at 13-plot-from-file.py to get a sense for how this works. Once
plotted, you should get a figure like this:

which is clearly the kind of thing you want to be able to do for like, any
kind of experiment data-throughput (as well as theoretical calculations!).
You can go even further, by calculating values that go into a graph one-
by-one. This is another common tool that’ll be used in calculations. Bring
up 14-plot-calculate.py. This program creates empty lists of points that
will be on the x and y axes, and then appends on 100 points of the sine curve
in a for loop. It then plots and shows them.

1.4.6 Changing features of graphs

If you want to change the axis limits, add labels, put in different colors, use
tick marks instead of lines, etc. then you can do all that (although unless
you’re trying to either publish or impress people, it’s not worth the time).
Bring up 15-plot-styles.py. The program produces

18
which again, looks fine. Note lines vs. dots are better than say, green vs.
red because they’re easier to distinguish in cases of noncolor printers and
colorblindness.

1.4.7 Scatter Plots

Data often isn’t of the form y(x), in which case we can observe it using
a scatter plot. There is a scatter function that lets us do this. Bring up
16-scatter.py. The file shows temperature vs. brightness of several stars
(a Hertzsprung-Russell diagram):

the main sequence is clearly visible, and the white dwarves are in the
5000K/2mag range, while the giants are the other clump around the 14mag
range. Many of the features that work for plot are the same as for scatter.

19
1.4.8 Density plot
Another useful plotting method is the density plot, in which we show data
values as a color or a brightness. To see this, bring up 17-density.py. This
program generates two plots,

where in the first the origin is at the upper left corner, and in the second
the origin is at the lower left corner. These are just conventions for plot
numbering (the data is the same). Some options for color display are:

• jet (default) – looks pretty bad

• hot (black-red-yellow-white)

• spectral (full spectrum)

• bone (gray scale w/ some blue)

• hsv (rainbow starting & ending with red)

• user-defined color schemes (google for websites that generate them)

Other options for imshow include extent, which changes the scale of data,
and aspect, which changes the aspect ratio. You can also limit the range of
data with xlim and ylim.

1.5 Lecture 5: wave interference, Mandelbrot set

In this lecture, we’ll see some examples of graphics with wave interference.
Dropping a pebble in water leads to waves radiating outward from where

20
the pebble was dropped. Given a drop location (x1 , y1 ), the distance from
this point is given by
p
r1 = (x − x1 )2 + (y − y1 )2

and the sine wave for height of the wave is

ξ1 (x, y) = ξ0 sin(kr1 )

where ξ0 is ampltiude and k = 2π/λ is wavevector with λ as wavelength.

Dropping a pebble at a different location would lead to

ξ2 (x, y) = ξ0 sin(kr2 )

and superposition of the waves would lead to

ξ(x, y) = ξ1 + ξ2 = ξ0 sin(kr1 ) + ξ0 sin(kr2 ).

Given λ = 5cm and ξ0 =1cm, and the centers of the circles being 20cm apart,
we can plot the resulting waves. Bring up 1-ripples.py. Plotting it yields

which honestly looks pretty cool. You can go one-better though, and that’s
by plotting fractals.

21
1.5.1 Plotting the Mandelbrot set
The Mandelbrot set is a fractal, containing structure within structure. Let
c be a complex number. Then for some z ∈ C, compute z 0 = z 2 + C. Keep
repeating the calculation, plugging in z 0 as z. The Mandelbrot set M is
defined as: let c ∈ C and z = 0, and iterate the above repeatedly. If |z| > 2,
then c ∈/ M . Otherwise c ∈ M .
Of course to get a complete picture of the set, we would need to iterate
this infinitely many times, but in practice we can just run it a finite number
of iterations. Since the program 1-1-mandelbrot.py isn’t available online
(it’s a HW problem from the textbook), and I want to get on to doing
interesting problems from the set, here’s a Mandelbrot image I made in
gnuplot:

1.5.2 Accuracy and Speed (Ch.4)

Computers have limitations, both for storage and processing time. We’ll
discuss these now, because they can seriously affect physics calculations.

Variables and ranges Python can’t store numbers that are arbitrarily
large. The largest value for a float is ≈ ± 10308 , and the real and imaginary
parts of complex numbers are limited by this too.
It’s possible to specify large numbers using “e” as an exponent, e.g., 2e9
means 2 × 109 . If the value of a variable exceeds its largest value, we say
it has overflowed. Python then sets it to a special value, inf , which treats
it as infinite. The smallest number is ≈ 10−308 , and if you go smaller the
number overflows, and the computer sets it to equal zero.

22
 For integers, Python uses an arbitrary size allowable by your computer.
As a comparison of using integers vs. floats for calculations, bring up
2-factorial-function.py. You’ll find that the integer factorial function
can compute 200!, while the float function overflows.

1.5.3 Numerical error

√
Numbers like π or 2 can only be represented approximately. The level of
precision for this in Python is 16 significant digits. For instance,
π = 3.1415926592653589793238... (1.6)
πPython = 3.1415926592653589793 (1.7)
diff = 2.38e − 17 (1.8)
and the difference is called rounding error. This means that numbers in a
floating point calculation are only guaranteed accuracy to 16 digits. This
means that you should never use if statements to test quality of floating
point numbers! Say you want to check x = 3.3. Do
e p s i l o n = 1 e −12
i f abs ( x −3.3)<e p s i l o n
print (x)

where you choose a value for  according to your situation.

So if we executed
from math import s q r t
x = sqrt (2)
√ √
we wouldn’t
√ get an exact x = 2. Rather, we would get x + ε = 2 or
x − ε = 2. We can’t necessarily rely on ε being small. So how do we model
this error?
Since numbers are accurate to 16 digits, then rounding error will typically
be of size x/1016 . Since we’re dividing up space into uniform intervals, the
simplest model that the error is a uniformly distributed random variable.
Recall then that the mean of a uniform √ random variable on [a, b] is (a + b)/2,
and that its variance is σ 2 = (b − a)/ 12. This means that the standard
deviation for a number x with 16 significant digits will be
σ = Cx where C ≈ 10−16
and this error can grow if we add, subtract, multiply, or divide numbers.
If we add two numbers, x1 + x2 , then the errors add. By assuming
independent errors, the standard deviations sum:
σ 2 = σ12 + σ22

23
which implies that q
σ=C x21 + x22
or, extended for addition of N numbers x1 , ..., xN :
N
X N
X
σ =
2
σi2 = C 2 x2i = C 2 N xˉ2
i=1 i=1

where xˉ2 is the mean-square value of the sequence.

√ This means that the
standard deviation on our numerical error goes as N . So it increases with
N , but reasonably slowly.

Example: numerical error

The code in 3-error.py,
from math import s q r t
x = 1.0
y = 1 . 0 + ( 1 e −14) ∗ s q r t ( 2 )
p r i n t ( ( 1 e14 ) ∗ ( y−x ) )
print ( sqrt (2) )

shows that these errors, if left unaccounted for, can cause problems. In this
case, the first print statement only gets sqrt(2) to a single decimal place of
accuracy.

Example: calculating derivatives

The mathematical definition of a derivation of f (x) is:

df (x) f (x + δ) − f (x)
= lim
dx δ→0 δ
from which we can get a reasonable approximation by making δ small. But
we can’t make it too small. For example, let f (x) = x(x − 1). We want the
value of the derivative at x = 1. Then

df (x) 
= 2x − 1|x=1 = 1.
dx x=1

We can also write a compute program to do this for us. The program
mentioned in the web-notes, 3-1-derivative.py, isn’t available .

24
1.5.4 Program Speed
Computers are limited in speed. However, they are fast, and can handle
a million operations easily. A billion might take minutes or hours, so the
author of the textbook suggests as a rough guide that your home laptop will
be able to handle a billion or less.
As an example, consider a quantum simple harmonic oscillator. It has
energy levels En = ~ω(n + 1/2) for n = 0, 1, 2, ..., and its average energy at
a temperature T will be
∞
1 X
hEi = En e−βEn
Z
n=0
P∞ −βEn
where β = 1/(kB T ) is inverse temperature, and Z = n=0 e is the
partition function.
Suppose we want to calculate hEi when kB T =100. Since we have an
exponential falloff with increasing n, we can get a reasonable approximation
by taking n large, say n = 1000. If we take units ~ = ω = 1, we can see
what this will look like in 4-qsho.py. There are some nice features of the
program to notice:
1. The constants are defined at the beginning

2. Only one for loop is used to calculate two sums

3. Even though e−βEn occurs in both sums, it is only calculated once

Note that we can increase the number of terms in the code to get more accu-
rate answers, but then there’s a trade-off between time and accuracy. Should
be able to estimate (based on 1million operations/s) how long calculations
take.

1.5.5 Matrix multiplication

Bring up 5-matrix-multiply.py. How do we multiply two matrices A, B
together? Right: “rows times columns”, in the sense

Cij = Aik Bkj .

One way to write this out explicitly is with 3 nested “for” loops. The inner-
most loop has N additions of multiplications, for a total of 2N operations.
The nested loops inside that each go around N times, giving a total of 2N 3
operations. So if N = 1000, that would be 2 × 109 operations. This is

25
doable, but if we were to try N = 2000, we would have 1.6 × 1010 , which is
a fair bit more difficult.
There is an algorithm called Strassen’s algorithm for matrix multiplica-
tion which brings the complexity down to N 2.8 , which can be much faster
for large matrices. The problem of figuring out an optimal exponent is still
open. The best known one is 2.3729, but the algorithm is only useful for
matrices that are too large to handle for modern computers (this algorithm
came from Francis le Gall, a quantum computational theorist).

26
Chapter 2

Lectures 6-10

We follow material available from Mark Wilde’s website, available here.

With lectures 1-5 and homework 1 done, we’re going to keep going.
A word of warning! The notes are going to get more personalized from
now on. It’s unlikely anyone aside from me is going to read these, so they’re
going to be foremost for my learning benefit. If that means extended tan-
gents into topics, or totally ignoring others because I feel confident about
them, then so be it.
Assessment on our three goals: (1) most of the physics problems we’ve
dealt with so far have been toy problems. Cherry-picking better problems
from the textbook should be an acceptable substitute for less interesting
homework problems. New rule: do at least one extra textbook problem per
homework set (for HW1: visualizing atoms in simple cubic lattice?). (2)
BaKoMa TeX seems to be working pretty well. Math typesetting (& lecture
typesetting) is frankly still too slow. I want to get down to 1hr/lecture.
Code block typesetting is good. More figure generation (especially hand-
drawn) is probably worthwhile, to see how well I can integrate that into the
notes. (3) most of the Python we’ve done so far is very simple. We haven’t
really touched anything that isn’t obvious vis-a-vis code structure. Reading
ahead on the web-notes should get a feel for where this is going and whether
it’s worth finishing the latter half of the course, after these 5 lectures.

2.1 Lecture 6: 3D plotting with visual package,

integral approximation
Installing VPython/VIDLE, which I did using the materials at vpython.org,
wound up requiring a downgrade to Python3.2.2. This is a bit of a pain,

27
 Figure 2.1: A cubic atomic lattice, say for NaCl.

but life goes on. Playing around in VPython, you can create images like the
one in Fig. 2.1, or the one in Fig. 2.2. Pretty cute.

But we can only spend so much time screwing around with VPython, so
we’re going to move on to bigger and better things.

2.2 Chapter 5: Integration (lec 6)

2.2.1 Trapezoidal rule
It isn’t always possible to evaluate integrals by hand. A better idea in these
cases is to evaluate them with a computer, to get approximate results. (E.g.,

28
 Figure 2.2: Points for the Portugal. the Man reference.

see “12 steps to computational fluid dynamics” online).

The simplest rule for doing this is the trapezoidal rule, implemented in
4-trapezoidal.py. Given a function f (x), suppose we want to integrate it
from x = a to x = b: Z b
I(a, b) = f (x)dx
a
do this by dividing the area under the curve into trapezoids. First divide
[a, b] into N slices, so that each slice has width h = (b − a)/N . Recall that
the area of a trapezoid of base h and sides c, d is (c + d)h/2.
The RHS of the k th slice occurs at a + kh. The LHS is at a + (k − 1)h.
Thus the area of the trapezoid for this slice is
1
Ak = h[f (a + (k − 1)h) + f (a + kh)]
2
so then the approximation for the integral is
N
X N
1 X
I(a, b) ≈ Ak = h [f (a + (k − 1)h) + f (a + kh)]
2
k=1 k=1

29
which means
1 1
I(a, b) = h[ f (a) + f (a + h) + f (a + 2h) + ... + f (a + (N − 1)h + f (b)]
2 2
" N −1
#
1 1 X
= h f (a) + f (b) + f (a + kh) (2.1)
2 2
k=1

which is the extended trapezoidal rule.

Example:
calculate the integral of x4 −2x+1 from x = 0 to x = 2, using the trapezoidal
rule. Analytic solving gives 4.4. Bring up 4-trapezoidal.py to try out
higher levels of accuracy. You’ll find for this interval that roughly 1,000
trapezoids gets you to within 10 −6 of the answer. More intervals is a trade
between computation time and accuracy.

2.2.2 Simpson’s Rule

The trapezoidal rule uses straight-line segments for approximations, but we
can use curves instead. Simpson’s rule fits intervals with quadratic curves.
It is implemented in 5-simpson.py.
We require three points to specify a quadratic (not just two, as with a
line). So the fit is of a quadratic to pairs of adjacent slices. Again, suppose
the integrand is f (x), and the spacing is h. Suppose the three points are
at −h, 0, +h. Then fitting a quadratic Ax2 + Bx + C through these points
gives

f (−h) = Ah2 − Bh + C
f (0) = C
f (h) = Ah2 + Bh + C
 
1 1 1
=⇒ A = f (−h) − f (0) + f (h)
h2 2 2
1
B = [f (h) − f (−h)]
2h
C = f (0)

and the area under the curve from −h to h is approximated by the area

30
under the quadratic, so
Z h
2
(Ax2 + Bx + C)dx = Ah3 + 2Ch
−h 3
1
= h[f (−h) + 4f (0) + f (h)]
3
which is Simpson’s rule. It approximates the area under two adjacent slices.
You need to have an even number of slices for this to work consistently, and
the approximate value of the integral comes out to
 
1  X X 
I(a, b) ≈ h  f (a) + f (b) + 4 f (a + kh) + 2 f (a + kh)
3  
k odd k even
1,...,N −1 2,...,N −2

This is implemented in 5-simpson.py. The two sums are calculated

separately, using range(1, N, 2) and range(2, N, 2) for their summation con-
ditions.

2.2.3 Quantifying error bounds

We can ask: what is the main source of error associated with the trape-
zoidal rule? The main source of error is not rounding error, but error in
approximating the integral, because we’re calculating an approximation to
the integrand. How big is this error? We can quantify it with some simple
tricks. Recall the trapezoidal rule gives

Z " N −1
#
b
1 1 X
f (x)dx = I(a, b) ≈ h f (a) + f (b) + f (a + kh)
a 2 2
k=1

and let xk = a + kh. Consider a slice of the integral between xk−1 and xk .
Perform a Taylor series expansion of f (x) about xk−1 , so that
(x − xk−1 )2 00
f (x) = f (xk−1 ) + (x − xk−1 )f 0 (xk−1 ) + f (xk−1 ) + ...
2
and integrate this from xk−1 to xk to get
Z xk
R xk
f (x)dx = f (xk−1 ) xk−1 dx +
xk−1
R xk
f 0 (xk−1 ) xk−1 dx(x − xk−1 ) +
f 00 (xk−1 ) R xk
2 xk−1 dx(x − xk−1 )2

31
and substitute u = x − xk−1 so that
Zxk
Rh
f (x) dx = f (xk−1 ) du
0
xk−1
Rh Rh
+f 0 (xk−1 ) 0 udu + 12 f 00 (xk−1 ) 0 u2 du + ...
(Christ those equations look terrible). If we keep slogging through this, and
I’m too lazy/not motivated enough to keep on type-setting all the math,
then we find the error to the trapezoidal rule is
1 2 0
ε= h [f (a) − f 0 (b)],
12
which is called the Euler-MacLaurin formula.
Summarizing, the trapezoidal rule is a first-order integration rule, accu-
rate up to terms proportional to h. The first order rule is accurate to O(h)
and has error O(h2 ).
You can similarly investigate the rounding error (floating point error) in
the trapezoidal rule case. Recall that it’s approximately Cx where x is the
value of the number, and C is machine precision of ≈ 10−16 . This means
that you can make h smaller, but there’s a limit to your increasing accuracy
due to accumulated rounding error. The happens when the approximate
error and the rounding error are equal, i.e., when
Z b
1 2 0 0
h [f (a) − f (b)] = C f (x)dx
12 a

which, solving for h, and setting h = (b − a)/N yields

s
f 0 (a) − f 0 (b) −1/2
N ≈ (b − a) Rb C
12 a f (x)dx

which means for all other factors order 1, N ≈ 108 . This is obviously large –
it would be wiser to use smaller N with more accurate rules like the Simpson
rule.

2.3 Lecture 7: Integration techniques

We can perform an error analysis analogous to the above for Simpson’s rule.
We find the approximate error to leading order is
1 4[f 000 (a)−f 000 (b)]
ε= h
90

32
 So Simpson’s rule is a third order integration rule, with fourth order
approximation
Rb error. The rounding error for Simpson’s rule is of order
C a f (x)dx so that
!1/4
f 000 (a) − f 000 (b)
N ≈ (b − a) Rb C −1/4
90 a f (x)dx

so the rounding error becomes important roughly when N ≈ 10, 000. There
is no point to using larger values of N with Simpson’s rule because you’ll
approach machine precision.
This also means that Simpson’s rule is not always ‘better’ than the trape-
zoidal rule. The function might be such that f 000 (a) is very large, and this
would make Simpson’s error larger than the trapezoidal error.
In many cases, we might not be able to compute derivatives of the func-
tion that we’re trying to integrate. Similarly, the function we’re trying to
integrate might not be a function, but it could be experimental data. How
do we estimate the error in this case? With a trick...
Suppose we’re using the trapezoidal rule for x = a to x = b and suppose
the number of steps in N1 , with step size h1 = b−a
N1 . The trick is to double
the number of steps, and then do the integral again. In other words, define
N2 = 2N1 , and h2 = (b − a)/N2 = h1 /2. Since the trapezoidal rule has error
O(h2 ), when we halve the slice size h, we quarter the size of the error. This
is good payoff, for the price of increasing complexity.
Suppose the true value of the integral is I. Denote the first estimate I1 ,
which has N1 steps. The difference between the true value and the estimate
is O(h2 ), so that we can write

I − I1 = ch21 =⇒ I = I1 + ch21

and then doing the same thing with the second estimate, write

I = I1 + ch21

which means that

I1 + ch21 = I2 + ch22 .
Then using that h2 = h1 /2, we get

I1 + c4h22 = I2 + ch22

=⇒ I2 − I1 = 3ch22

33
so this implies that the error on the 2 nd estimate is
1
ε2 = ch22 = (I2 − I1 )
3
So we get a simple way to estimate errors while avoiding the Euler-Maclaurin
formula. We can use the same idea to get an estimate for Simpson’s rule of
1
ε2 = (I2 − I1 ).
15

2.3.1 Choosing the Number of Steps for an Integral

Often we’d like to calculate to some desired accuracy, and we’d like to know
how many steps are needed to achieve that accuracy. In other words, we
want to use as many steps needed to get to machine precision.
A simple method to do this is as follows: stat with a given N and
keep doubling it until you reach the desired accuracy. Use the formula
ε2 = 13 (I2 − I1 ) to figure out the error. The error on the ith step is given
by 13 (Ii − Ii−1 ). A nice feature is that when doubling the number of steps,
there’s no need to recompute, just sample more and add them. For the
trapezoidal rule, the initial sample points and coefficients are given by

and the new ones are then

where sample points for the first estimate are nested inside those for the
second estimate (n.b., hand-drawn in OneNote, copy-pasted into BaKoMa
TeX. Nice... multiple colors and you can go further).
To make this precise, consider the ith iteration of the trapezoidal rule.
Let Ni be the number of slices, and let hi = (b − a)/Ni be the width of each

34
slice. Then Ni−1 = Ni /2 of hi−1 = 2hi , and
" #
N
X i −1
1 1
Ii = hi f (a) + f (b) + f (a + khi )
2 2
k=1
 
1 1 X X 
= hi 
 2 f (a) + 2 f (b) + f (a + khi ) + f (a + khi )

k even k odd
2,...,Ni −2 1,...,Ni −1

Now consider that

Ni /2−1
X X
f (a + khi ) = f (a + 2khi )
k even k=1
2,...,Ni −2
Ni−1
X
= f (a + khi−1 ) (Check upper bound)
k=1

which means that

" #
N
X i −1 X
1 1 1
Ii = hi−1 f (a) + f (b) + f (a + khi ) + hi f (a + khi )
2 2 2
k=1 k odd
1,...,Ni −1
1 X
Ii = Ii−1 + hi f (a + khi ) (F)
|2 {z } k odd
old estimate from 1,...,Ni −1
i−1 iteration | {z }
new estimate

This means that we can compute adaptively, and have nearly the same
computational cost as if we had proceeded non-adaptively. The algorithm
is as follows:

1. Choose initial number of steps, N1 . Decide on desired target accuracy.

Calculate first approximation I1 using N1 and the trapzeoidal rule. Set
i = 2.

2. Double the number of steps, and use (F) to calculate Ii . Calculate

the error using εi = 13 (Ii − Ii−1 ) .

3. If |εi | < desired accuracy, stop. Otherwise, i = i + 1, and go to 2.

35
There is a similar formulation for the Simpson rule as well. Upon doubling
the number of steps, we have
1
εi = (Ii − Ii−1 )
15
and then the iteration rule involves

 
1 X 
si = f (a) + f (b) + 2 f (a + khi )
3 
k even
2,...,Ni −2
2 X
Ti = f (a + khi )
3
k odd
1,...,Ni −1

so that

si =si−1 + Ti−1
Ii = hi (si + 2Ti ).

Then the algorithm for Simpson’s rule is:

1. Choose an initial number of steps and a desired target accuracy. Cal-

culate S1 , T1 , I1 . Set i = 2.

2. Double the number of steps. Calculate Si , Ti , Ii . Calculate error as

εi = 15
1
(Ii − Ii−1 ).

3. If |εi | < desired target accuracy, then stop. Else, step i = i + 1 and go
to 2.

2.3.2 Romberg Integration

We can actually do better than the adaptive method of last section. Recall
that the leading order error the trapezoidal rule is
1
ch2i = (Ii − Ii−1 )
3
and the true value of the integral is then

I = Ii + ch2i + O(h4i )

36
where O(h4i ) is the next term in the series (recall that there are only even-
order terms in the series). So then
1
I = Ii + (Ii − Ii−1 ) + O(h4i )
3
is now an accurate expression to third order, and has fourth-order error. We
can take this process further. This is done in very close detail in pages13-16
of Mark Wilde’s notes.
Summarizing: it is possible to eliminate the O(h4i ) term to get an ex-
pression accurate to 5th order, with 6th order error. This process can be
continued, cancelling out higher-order terms and getting more accurate re-
sults. The whole thing can be generalized in a method called Romberg
integration. The computational complexity is essentially the same as the
trapezoidal rule (n.b., computational complexity doesn’t mean ‘ease of writ-
ing the code’ – it means number of operations it takes for a computer to
calculate the integral estimate), while being much more accurate.

2.4 Lecture 8: Higher-order integration methods,

and Gaussian quadrature
The trapezoidal rule is based on straight line segments, while the Simpson
rule is based on quadratics. We can use higher-order rules, fitting f (x) with
cubics, quartics, etc. The general form of these rules is

Z b N
X
f (x)dx ≈ wk f (xk )
a k=1

where xk are positions of sample points and wk are weights. For the trape-
zoidal rule, xk are continuously spacing, and the first and last weights are
1/2. All weights in between are 1. The Simpson rule has weights 1/3 for
the first and last points, and the weights between alternate between 4/3 and
2/3.
Higher order rules have the same idea: fit the curve to some polynomial,
integrate it to get the weights wk , and multiply these weights by samples
f (xk ) to approximate the integral.
Here is a table of the weights up to quartic order:

37
 Degree Polynomial Coefficients
1 line 1/2, 1, 1, ... , 1, 1/2
2 quadratic 1/3, 4/3, 2/3, 4/3, ..., 1/3
3 cubic 3/8, 9/8, 9/8, 3/4, 9/8, 9/8, 3/4, ...
4 quartic 14/45, 64/45, 8/15, 64/45
and they just kind of keep going. These higher order integration rules
are called Newton-Cotes formulas.

We can do better... higher-order polynomials

The trapezoidal rule is exact for straight lines. The Simpson rule is exact
for quadratics. If we have N sample points, then we can just fit an N − 1
degree polynomial to the whole interval. The resulting method will be exact
for N − 1 or below degree polynomials.

We can do even better! Non-uniform spacing

We’ve been using uniformly spaced sample points. The advantages of using
these are that

1. They’re easiest to program

2. Can increase the number of points by in-between points

but we can also use non-uniformly spaced sample points. The main advan-
tage of this is that they can give accurate answers, with a small number of
sample points.
So let’s allow for the possibility to vary not only weights, but also sample
points. This is now 2N degrees of freedom, and suggests that we could have
an integration rule exact for polynomials of order 2N − 1. The method to
do this is called Gauss Quadratures.

2.4.1 Gauss Quadratures

Our approach is as follows:

1. Derive integration rules w/ non-uniform sample points.

2. Then choose an optimal set of non-uniform sample points.

38
 For 1), suppose our non-uniform sample points are {xk }N
k=1 , and we
want an integration rule of the form
Z b N
X
f (x)dx ≈ wk f (xk ).
a k=1

Then we need to choose our weights wk for a general f (x). We use the
method of interpolating polynomials (Lagrange polynomials):
Y x − xm
φk (x) =
x k − xm
m=1,...,N
m6=k
x − x1 x − xk−1 x − xk+1 x − xN
= × ... × × × ... ×
x k − x1 xk − xk−1 xk − xk+1 x k − xN
so there’s one factor for each sample point except xk . So φk (x) is a polyno-
mial in x of degree N − 1. And there are N different φk (x) functions for N
different sample points. Then also note that
φk (xm ) = δm,k
and consider the function
N
X
Φ(x) = f (xk )φk (x).
k=1

This is a polynomial of degree N − 1, and evaluating it at any sample points

gives
N
X N
X
Φ(xm ) = f (xk )φk (xm ) = f (xk )δkm = f (xm ).
k=1 k=1

So Φ(x) fits f (x) at all sample points. It is also the unique polynomial which
does so. There are N coefficients and N constants, so this is the one.
To get an approximation, just integrate Φ(x) from a to b:
Z Z b
f (x)dx ≈ Φ(x)dx
a
Z b "XN
#
= f (xk )φk (x) dx
a k=1
N
X Z b
= f (xk ) φk (x)dx,
k=1 a

39
so the weights are given by
Z b
wk = φk (x)dx. (2.2)
a

Unfortunately, there’s no closed-form formula for Eq.(2.2), so we need to

approximate it with Romberg integration or the Simpson rule.
The point is that we calculate the weights wk once, and then use them
to integrate a LOT of functions. Another point is that you can calculate the
weights for a particular set of sample points (& their domain of integration)
and then map to other points. The standard interval for this isR [−1, 1].
1
So give a set of sample points xk ∈ [−1, ] and weights wk = −1 φk (x)dx,
we translate [−1, 1] to [a, b] and stretch. The rule for mapping xk ∈ [−1, 1]
to [a, b] is
1 1
x0k = (b − a)xk + (b + a).
2 2
Interpretation: translate to the midpoint of [a, b] and then stretch out to
the full size of [a, b]. If the width of the interval changes, then rescale by
1
wk0 = (b − a)wk .
2
After doing this, the integral approximation will be
Z b n
X
f (x)dx ≈ wk0 f (x0k ).
a k=1

Great! Except we’ve only solved half the problem. Now we need to figure
out where to place the sample points. Doing this is based on the mathematics
of Legendre polynomials. Recall that our goal is to pick sample points such
that integration is exact for polynomials of degree 2N − 1 or less.

Grid selection; Legendre polynomials

The Legendre polynomial PN (x) is an N th order polynomial in x with the
property that
Z 1
xk PN (x)dx = 0, ∀k ∈ {0, ..., N − 1},
−1

so it’s orthogonal to all polynomials with degree N −1 or less. It also satisfies

the normalization Z 1
2
[PN (x)]2 dx = .
1 2N + 1

40
 So how do we find these polynomials? Well P0 (x) = const. and the
normalization imply that P0 (x)=1. Then P1 (x) is a first order polynomial,
ax + b, satisfying Z 1
x0 (ax + b)dx = 0,
−1
so computing the integral gives
1
ax2 
+ bx = 2b =⇒ b = 0
2 −1

and the normalization gives a = 1, so P1 (x) = x. We can then use the

(non-obvious) recurrence relation,

(N + 1)PN +1 (x) = (2N + 1)xPN (x) − N PN −1 (x)

to get the higher order terms. You can find them online.
So then suppose q(x) is a polynomial of degree less than N , so that
N
X −1
q(x) = c k xk
k=0

then
Z 1 N
X −1 Z 1
q(x)PN (x)dx = ck xk PN (x)dx = 0.
−1 k=0 −1

Another property of the Legendre polynomials: for all N , PN (x) has N

real roots that all lie in [−1, 1]. So there are N values of x in [−1, 1] such
that PN (x) = 0.
Then suppose that f (x) is a polynomial in x of degree 2N − 1 or less.
We can divide f (x) by PN (x). Do you remember polynomial division from
highschool? (nope.) This leaves us with

f (x) = q(x)PN (x) + r(x)

where q(x), r(x) are polynomials of degree N − 1 or less.

So then we can write the integral as
Z 1 Z 1 Z 1
f (x)dx = q(x)PN (x)dx + r(x)dx
−1 −1 −1
Z 1
= r(x)dx
−1

41
so we only need to find the integral of a polynomial of degree N − 1 or less.
But we’ve already done this! For any choice of sample points xk , a poly-
nomial of degree N − 1 or less can be fitted using interpolating polynomials
φk (x), so we have
Z 1 Z 1 N
X
f (x)dx = r(x)dx = wk r(xk )
−1 −1 k=1
R1
where wk = −1 φk (x)dx. This is equal is exact. So the method for integrat-
ing any polynomial of degree 2N − 1 or less over [−1, 1] is as follows:
1. Divide by the Legendre polynomial PN (x) to get r(x), the remainder
polynomial.

2. Then integrate r(x) using any N sample along with its corresponding
weights.

Picking arbitrary grid points

We can be slightly more clever by picking sample points to be any that we
wish. Consider that
N
X N
X N
X
wk f (xk ) = wk q(xk )PN (xk ) + wk r(xk ).
k=1 k=1 k=1

Recall then that PN (x) has N zeros between -1 and 1, so choose N sample
points to be positions of these zeros.
N
X N
X
=⇒ wk f (xk ) = wk r(xk ),
k=1 k=1

since
Z 1 Z 1 N
X
f (x)dx = r(x)dx = wk r(xk )
−1 −1 k=1
Z 1 XN
= f (x)dx = wk f (xk ).
−1 k=1

Thus we have an integration rule that allows for integrating any poly-
nomial f (x) with order 2N − 1 or less from [−1, 1], and we get an exact
answer, up to rounding error! This is done using only N sample points.

42
2.5 Lecture 9: Choosing integration methods; in-
tegrals for infinite ranges; derivatives
When choosing an integration method from (1) the trapezoidal rule, (2)
Simpson’s rule, (3) Romberg integration, or (4) Gaussian quadrature, there
are some guidelines that apply.
Generally, higher order methods work better for smooth functions. If
not, then it’ll be better to use simpler methods, because variation in data
won’t be reflected at the sample points. The trapezoidal rule is good for inte-
grating data from experiments at uniformly spaced sample points. It’s good
for poorly behaved functions. Simpson’s rule relies on a higher order ap-
proximation of integrands in order to be accurate. And Gaussian quadrature
is very accurate, but not desirable if you require uniformly spaced sample
points.

2.5.1 Computing integrals over infinite ranges

How do you compute integrals like
Z ∞
f (x)dx ?
0
R∞
The simple idea is to use change of variables. For 0 , the most common
change is z = 1+x
x
, or x = 1−z
z
, which changes the interval to [0, 1]. Then

dx 1 dz
= =⇒ dx = ,
dz (1 − z) 2 (1 − z)2
and Z Z  
∞ 1
z 1
f (x)dx = f dz
0 0 1−z (1 − z)2
and just use standard techniques. This is the standard change of variables,
but not the only possible one. For instance, for the gamma function,
Z ∞
Γ(a) = xa−1 e−x dx,
0

this change isn’t too good. In this case, most of the area of the curve falls
under the maximum at a − 1, but the change puts the peak near the edge of
[0, 1] and uniformly spaced samples won’t do well. So we can use a different
transformation: z = c+x x
. Picking c = a − 1 for the gamma function then
xγ
places the peak at z = 1/2. Other choices include z = 1+x γ , for some γ.

43
 To do the integral from a to ∞, the approach is similar. First translate,
and then scale. In other words, let y = x − a, and
y x−a z
z= , or z = =⇒ x = + a.
1+y 1+x−a 1−z

In this case, dx = dz
(1−z)2
, and then
Z Z
∞ 1
1 z
f (x)dx = f( + a)dz.
a 0 (1 − z)2 1 − z

Going from −∞ to a, just substitute z = −z. For integrals going from

−∞ to ∞ it’s possible to just split them in two. You can also just do a
single change of variables, e.g.,

z 1 + z2
x= , dx = dz
1 − z2 (1 − z 2 )2
Z Z
∞ 1
1 + z2 z
→ f (x)dx = f( )dz.
−1 (1 − z ) 1 − z2
2 2
−∞

Another possibility is
dz
x = tan z, dx =
cos2 z
Z ∞ Z π/2
f (tan z)
→ f (x)dx = )dz.
−π/2 cos z
2
−∞

R∞ 2
Example: Calculate I = 0 e−t dt. Make the variable change z = t
1+t ,
then we get
Z 1 2 2
e−z /(1−z)
I= dz.
0 (1 − z)2
Bring up infint.py from Mark Newman’s page:
from gaussxw import gaussxwab
from math import exp

def f ( z ) :
r e t u r n exp(−z ∗∗2/(1 − z ) ∗ ∗ 2 ) /(1− z ) ∗∗2

N = 50
a = 0.0
b = 1.0
x ,w = gaussxwab (N, a , b )

44
s = 0.0
f o r k i n r a n g e (N) :
s += w [ k ] ∗ f ( x [ k ] )
print ( s )
√
the value of the actual integral is π/2, and Gaussian quadrature is accurate
to machine precision. (!!!)

2.5.2 Multiple Integrals

Suppose the integral you want to evaluate is
Z 1Z 1
I= f (x, y)dxdy.
0 0
R1
Rewriting in terms of F (y) = 0f (x, y)dx, this is just
Z 1
I= F (y)dy.
0
Then approximate F (y) at some points y, and integrate. Gaussian quadra-
ture would suggest to do
XN
I≈ wj F (yi )
j=1
for some weights wj and sample points yi , and then
N
X
F (y) ≈ wi f (xi , y)
i=1
can substitute in to give the final approximation as
N X
X N
I≈ wi wj f (xi , yj ).
i=1 j=1

There’s no reason that the sample points (xi , yj ) need to be on a grid. We

could attempt to find the ‘best fit’ as with Gaussian quadratures, but there is
no answer known here. We could also select the sample points RANDOMLY.
This is called Monte Carlo integration, and we’ll talk about it later. Before
we get there, we can complicate the multiple-integral case by considering
Z 1 Z y
I= dy dx f (x, y),
0 0
Ry R1
then define F (y) = 0 f (x, y)dx, and I = 0 F (y)dy. If we use the same
method as before, the same points are crammed in near each other in certain
regions.

45
2.5.3 Derivatives (finite difference and more)
Recall the definition of a derivative as a slope:
df f (x + h) − f (x)
= lim .
dx h→0 h
The basic idea is we want to implement this formula, but we can’t pass to
the case of a ‘limit’. So instead make h small, and approximate as
df f (x + h) − f (x)
≈ .
dx h
This is called a forward difference. A backward difference is then
df f (x) − f (x − h)
≈ .
dx h
There’s no real reason to pick one over the other (it turns out, they’re
both not actually very accurate). What are the errors in approximating
derivatives this way?

• Rounding error (discussed before)

• Approximation error, since h → 0 is impossible.

To quantify these errors, we’ll use Taylor expansions. Assume the func-
tion is ‘smooth’, i.e., infinitely differentiable. Then
1
f (x + h) = f (x) + hf 0 (x) + h2 f 00 (x) + ...
2
which we can rearrange to get
f (x + h) − f (x) 1 00
f 0 (x) = − hf (x) + ...
h 2
The forward difference here only gives the first term. To leading error, the
approximation error will be 12 h|f 00 (x)|.
But the problem now is that subtracting numbers can lead to large
rounding errors if h is very small. Recall that floating point precision is
good to 16 significant figures. If h winds up being past those figures (quite
likely, since we’re taking a small h limit), we will hit large rounding errors.
So there is a trade-off between small h and rounding error.
Recall that computers can calculate f (x) to accuracy Cf (x) for C ≈
10 . Given that f (x + h) is usually close to f (x), the absolute value on
−16

46
rounding error of f (x + h) − f (x) in the worst case will be 2C|f (x)|. So the
worst case rounding error on f (x+h)−f h
(x)
is 2C|fh(x)| . Also, the approximation
error is 12 h|f 00 (x)|. So the total error is

2C|f (x)| 1
ε= + h|f 00 (x)|.
h 2
We want to find h that minimizes this error. Take dε/dh = 0, giving

−2C|f (x)| 1 00
0= + |f (x)|
s h
2 2

 f (x) 
=⇒ h = 4C  00 .
f (x) 

Plugging back in for ε gives

p
ε= 4C|f (x)f 00 (x)|.

So if f (x) and f 00 (x) are of order 1, then final error will be ≈ 10−8 .

Improving difference methods: central difference

df f (x + h/2) − f (x − h/2)
≈ .
dx h
The sample points are still h apart. But this reduces approximation error!
1 1 1
f (x + h/2) = f (x) + hf 0 (x) + h2 f 00 (x) + h3 f 000 (x) + ...
2 8 48
1 0 1 2 00 1
f (x − h/2) = f (x) − hf (x) + h f (x) − h3 f 000 (x) + ...
2 8 48
f (x + h/2) − f (x − h/2) 1
=⇒ f 0 (x) = − h2 f 000 (x) + ...
h 24
so we cancelled the f 00 (x) term, and the new approximation error to leading
order is h2 |f 000 (x)|/24. Similar to before, the total error comes out to

2C|f (x)| 1
ε= + h2 |f 000 (x)|.
h 24
Differentiating with respect to h and nulling yields
  
 f (x)  1/3
h= 
24C  000  ,
f (x) 

47
and substituting back in gives a minimum error of
 1/3
9 2 2 000
ε= C [f (x)] |f (x)| .
8
So with f (x) and f 000 (x) of order 1, we’re taking h ≈ 10−5 , with total error
≈ 10−10 . This is better than before.

Taking derivatives of sampled functions

Suppose we have a sampled function at evenly spaced intervals, a distance
h apart. If we use central differences, then we use points at x + h and x − h.
The formula is now
df f (x + h) − f (x − h)
≈ .
dx 2h
In this case, does central or forward difference give the more accurate an-
swer? It depends...
The total error for central difference with h → 2h is h2 |f 000 (x)| while that
for forward differences is h|f 00 (x)|, so we would need h < |f 00 (x)/f 000 (x)|.
However, we could calculate the derivative at points in between samples,
and get a better answer, because we would have x + h/2 and x − h/2.

2.6 Lecture 10: Higher order derivatives, interpo-

lations, Gaussian elimination
The techniques described in Lecture 9 approximate the derivative with a
straight line. This is a first-order approximation. Can we do better with
higher-order approximations? This worked for integrals, what about deriva-
tives?
Consider fitting a quadratic to a function of interest that we would like to
differentiate. Suppose we want the derivative at x = 0. As in the derivation
for Simpson’s rule, we require a quadratic ax2 + bx + c to fit our function,
and so
f (−h) = ah2 − bh + c,
f (h) = ah2 + bh + c,
c = f (0).
We could then solve for a, b, c, but there’s actually no need to. Given y =
ax2 + bx + c, 
dy 
= [2ax + b]|x=0 = b.
dx x=0

48
So we just need b, and we can get it by subtracting two of the equations in
our quadratic fit:

f (h) − f (−h) = 2bh

f (h) − f (−h)
=⇒ b = .
2h
But this is just the central difference! If we repeated this for other values of
x, we would find
df f (x + h) − f (x − h)
≈ .
dx 2h
So it is really just central differences.
However, going to higher orders helps. There is also a distinction between
even and odd orders. For all of the following, sample points need to be
chosen to be symmetric about x. For odd orders, sample points fall halfway,
e.g., ..., −3h/2, −h/2, h/2, 3h/2, ..., while for even orders, they fall at integer
points, e.g., ..., −2h, −h, 0, h, 2h, ..., since they have an odd number of points
for the approximation.
N.b. none of the figures alluded to in lecs8/9/10 are available
online.
It turns out, for the nth approximation we can get it exact if the function
is a polynomial of degree n or less. The order of the error is given in a table
in the textbook.

Second derivative of a function

Just apply the first derivative approximation twice (i.e., central difference).

f (x + h) − f (x)
f 0 (x + h/2) ≈
h
0 f (x) − f (x − h)
f (x − h/2) ≈ ,
h
so then f 00 (x) = [f 0 (x + h/2) − f 0 (x − h/2)]/h. Substituting in,

f (x + h) − f (x) − [f (x) − f (x − h)]

f 00 (x) =
h2
f (x + h) − 2f (x) + f (x − h)
=
h2

49
and we can use this to help solve second-order differential equations. How
do we estimate the error? Again, using Taylor expansions:
1 1 3 000 1 4 0000
f (x + h) = f (x) + hf 0 (x) + h2 f 00 (x) + h f (x) + h f (x) + ...
2 6 24
1 1 3 000 1 4 0000
f (x − h) = f (x) − hf 0 (x) + h2 f 00 (x) − h f (x) + h f (x) − ...
2 6 24
Adding and rearranging, we find

f (x + h) − 2f (x) + f (x − h) 1
f 00 (x) = − h2 f 0000 (x) + ...
h 2 12
The first term is approximate, and the (absolute value of the) next
term is the leading order error. Rounding error is now (in the worst case)
4C|f (x)|/h2 , so the total error is

4C|f (x)| 1
ε= + h2 |f 0000 (x)|.
h 2 12
Take dε/dh = 0, and get
  
 f (x)  1/4
h= 48C  0000  .
f (x) 

This means that the optimal error size will be

1
ε = h2 |f 0000 (x)|
6
 1/2
4
= C|f (x)f 0000 (x)|
3

which means ε ≈ 10−8 , about the same as forward and backward difference
methods.

Partial derivatives:
Done in the obvious way,

∂f f (x + h/2, y) − f (x − h/2, y)
≈
∂x h
∂f f (x, y + h/2) − f (x, y − h/2)
≈
∂y h

50
Derivatives of noisy data:
Taking derivatives of noisy data amplifies the noise because of sharp transi-
tions. Ways around this include: (1) increasing h to eliminate fluctuations,
and (2) fitting a curve, and then differentiating the curve (e.g., a least-
squares fit).

2.6.1 Interpolation
Methods for analyzing interpolation are similar to what we’ve already used.
Interpolation asks the question: given a value of f (x) at a and b, what is the
value in between? The first obvious method is linear interpolation. Draw a
straight line between the points, with slope

f (b) − f (a)
m= .
b−a
In between then, the distance y is y = m(x − a), z = f (a). We can approxi-
mate
(b − x)f (a) + (x − a)f (b)
f (x) ≈ .
b−a
How accurate is this formula? Once again, we resort to Taylor expansions
(you might be noticing a trend by now).
1
f (a) = f (x) + (a − x)f 0 (x) + (a − x)2 f 00 (x) + ...
2
1
f (b) = f (x) + (b − x)f 0 (x) + (b − x)2 f 00 (x) + ...,
2
so when we substitute we find
(b − x)f (a) + (x − a)f (b)
f (x) = + (a − x)(b − x)f 00 (x) +...
b − a | {z }
| {z } leading order error
interpolation formula

Thus if f 00 (x) varies slowly, then the largest error is in the middle of the
interval. If the width of the interval is h = b − a, then being in the middle
means that x − a = b − x = h/2, and the leading order error is h2 |f 00 (x)|/4.
The worst-case error is then O(h2 ). Note that we need not be too careful
with rounding error inside the interval because we have the sum of f (a) and
f (b).

51
2.6.2 Chapter 6! Solving linear equations
Note these lecture notes get extra ‘value’ now in the sense that Newman’s
book is now closed. I guess you gotta put bread on the table.
Linear equations have the form Ax = v. Algorithms for matrix inversion
are costly, so it’s more desirable to use Gaussian elimination. Suppose

2w + x + 4y + z = −4
3w + 4x − y − z = 3
w − 4x + y + 5z = 9
2w − 2x + y + 3z = 7.

We can rewrite this as a matrix equation:

    
2 1 4 1 w −4
3 4 −1 −1 x  3 
   =  .
1 −4 1 5  y   9 
2 −2 1 3 z 7
Notice that:
1. We can multiply any row by a constant and the eqn/soln doesn’t
change
2. We can take any linear combination of two equations to get another
equation which is consistent with the original solution.
So we have some moves. How do we solve our set? For this particular set
of equations: (1) divide row 1 (1-based counting) by top-left element, (2)
subtract 3 times 1st row from 2nd, (3) repeat trick for 3rd and 4th row,
yielding     
1 0.5 2 0.5 w −2
0 2.5 −7 −2.5  x   9 
   =  .
0 −4.5 1 4.5   y   11 
0 −3 −3 2 z 11
From this point, we can repeat the algorithm, but for the second row, and
eventually the third, finally getting
    
1 0.5 2 0.5 w −2
0 1 −2.8 −1  x  3.6
   =  .
0 0 1 0   y  −2
0 0 0 1 z 1

52
Since the 4×4 matrix is now upper-triangular, we can solve by back substi-
tution.

Back subsitution
Starting from     
1 a01 a02 a03 w v0
0 1 a12 a13   x  v1 
   =  ,
0 0 1 a23   y  v2 
0 0 0 1 z v3
we get the equations

w + a01 x + a02 y + a03 z = v0 ,

x + a12 y + a13 z = v1 ,
y + a23 z = v2 ,
z = v3 .

So start from the last and back substitute. If it were published online, we
could now check out 5-guasselim.py, but it’s pretty self-explanatory.

Pivoting
What if the equation system has a00 = 0? Just permute rows. The best idea
is to use partial pivoting, in which you find the entry farthest from zero, and
swap that one in. This will lead to the smallest rounding error.

53
Chapter 3

Lectures 11-15

We continue with the material from Mark Wilde’s website, here. We’ve
made it through the first ten lectures, as well as the first two homeworks.
We’re a little under the half-way mark (since there are five homeworks; 20
lectures).
The material’s going to get more and more advanced now. We’re going
to dive into numerical methods. These next lectures will cover linear alge-
bra operations (solving linear equations, inverting matrices), optimization
methods, and at the end, Fourier transforms. A lot of the optimization ma-
terial will be similar to what I’ve already seen at USC, but since it’ll be with
a focus towards designing actual programs (in lieu of the theory) it should
be worth going through.
Our goals at the start were (1) learn how to approach and solve physics
problems with computers, (2) develop fluency with BaKoMa TeX, and (3)
develop fluency with Python. On (1) we haven’t been solving problems as
much as we’ve been developing tools that help us understand them. The
Feigenbaum plot was a great exercise in data visualization as well as vec-
torizing code, and methods for prime number generation and recursive defi-
nitions are good to know. Gauss quadrature is probably most applicable to
the physics side, since we’ll be using it to evaluate integrals left and right.
On (2) BaKoMa is getting better, especially with new keybindings and user-
defined macros. Dealing with matrices will be a challenge in the next part,
and might mean more figure integration. On (3) I’m beginning to get a feel-
ing for why Numpy, Scipy, and Matplotlib together make Python a really
powerful tool for scientific computation.
We’re committing here to at least HW4, because those problems look
awesome.

54
3.1 Lecture 11: LU decomposition
Last time we talked about Gaussian elimination for solving linear systems
of equations. Now we’ll talk about LU decomposition.
Often we want to solve equations of the form Ax = v for the same A but
different vectors v. We would like to do this in a way such that we don’t
have to solve Gaussian elimination every time. For this, we can use the LU
decomposition. Let  
a00 a01 . . . a03
 .. .. 
A= .
 . 

 
a30 . . . a33
In Gaussian elimination, we first divide out the first row by a00 . Then use
the entry there to cancel out the first entries on the other rows. This can
be written as a lower-triangular matrix:
 
1 0 0 0
1   −a10 a00 0 0  = L0
a00 −a20 0 a00 0 

−a30 0 0 a00

Left multiplying L0 A yields

 
1 b01 b02 b03
 0 b11 b12 b13 
 
 0 b21 b22 b23 
0 b31 b32 b33

the next step is then to divide out the second row by b11 , and subtract from
the lower rows. We can write this as a matrix too:

55
and then L1 times L0 A gives

We can continue this process to get two more matrices L2 , L3 . We would

then have
L3 L2 L1 L0 A x = L3 L2 L1 L0 v.
| {z } | {z }
upper triangular all known
matrix
From which point we can use back substitution to solve. The advantage of
this method is that we can use L3 ...L0 on any vectors v.
In practice though, things are done slightly differently. Let L = L−1 −1 −1 −1
0 L 1 L 2 L3 ,
and U = L3 L2 L1 L0 A, A = LU . Then since Ax = v, =⇒ LU x = v. It
turns out L actually has a simple form. Consider

Then

and there are similar forms for L−1 −1 −1

1 , L2 , L3 so that

56
 .

Once we have an LU decomposition, we can solve Ax = v directly as

Nice. So A = LU =⇒ Ax = LU x = v. Define y as y = U x, then Ly = v.

From the LHS of Ly = v, we can back substitute, figure out y, and then use
U x = y and back substitution to get x. To avoid issues with small or zero
elements, use pivoting or partial pivoting. Then the process breaks down
into finding the LU decomposition, and two back substitutions.
To actually solve systems of linear equations in Python, just use
from numpy . l i n a l g import s o l v e
x = s o l v e (A, v )

3.1.1 Calculating matrix inverses

We can use what we already know to solve systems of the form AX = V for
matrix X and matrix V . To do this, do each column separately. Then if we
set V = I, we are solving for the inverse of A.

Tridiagonal and bounded matrices

Consider

57
a tri-diagonal matrix. Gaussian elimination works well for this, but there’s
no need to do it in full. Just subtract each row from the one immediately
below it. There is a similar kind of procedure for bounded matrices.

Example We are given N identical masses connected by identical springs,

Let ξi denote the displacement of the ith mass relative to rest position. Then
Newton’s second law is that
d2 ξ i
m = k(ξi+1 − ξi + k(ξi−1 − ξi ) + Fi
dt2
where Fi is any external force on mass i. The endpoint masses are described
by
d2 ξ 1
m 2 = k(ξ2 − ξ1 ) + F1 ,
dt

d2 ξN
m = k(ξN −1 − ξN ) + FN .
dt2
We want to derive the system with a periodic force applied to the first mass.

F1 = Ceiωt ,

where C is some constant. The net result of the applied force will be to
make masses oscillate with angular frequency ω so that ξi (t) = xi eiωt . We
can substitute in to find

−mω 2 x1 = k(x2 − x1 ) + C
−mω 2 xi = k(xi+1 − xi ) + k(xi−1 − xi )
−mω 2 xN = k(xN −1 − xN ).

We can rearrange these as

(α − k)x1 − kx2 = C
αxi − kxi−1 − kxi+1 = 0
(α − k)xN − kxN −1 = 0,

where α = 2k − mω 2 . In matrix form, this looks like

58
Bringing up 1-springs.py, we can(’t; not available) see that some of the
masses move vigorously, while others do not.

3.1.2 Eigenvalues and eigenvectors: QR algorithm

Have a number of important applications in physics (e.g., the energy spectra
of Hamiltonians in quantum mechanics). Let A be a symmetric matrix. The
eigenvector v is such that
Av = λv,
for scalar λ. A N × N matrix has N eigenvectors and N eigenvalues. Eigen-
vectors are orthonormal to each other. We can thus take eigenvectors {vi }
to be columns of a single matrix V and write all equations as
Avi = λi vi .
As a single matrix equation, this would be
AV = V Λ,
for Λ the diagonal matrix of eigenvalues. V is an orthogonal matrix, so
V T V = V V T = I.
The most widely used technique for getting eigenvectors and eigenvalues is
called the QR algorithm. How does it work?
First, calculate the QR decomposition of a matrix as
A = QR
where Q is orthogonal and R is upper triangular. Any square matrix A can
be written like this. The first step of the algorithm is
A = Q1 R 1 .

59
Multiply by QT1 on the left to get

QT1 A = QT1 Q1 R1 = R1 .

Now let A1 = R1 Q1 , which is the reverse of A. Then

A1 = R1 Q1 = QT1 AQ1 .

Then we iterate this process. We find the QR decomposition of A1 as A1 =

Q2 R2 . Define

A2 ≡ R2 Q2 = QT2 A1 Q2 = QT2 QT1 AQ1 Q2 .

Continuing, we get

A3 = QT3 QT2 QT1 AQ1 Q2 Q3

..
.
Ak = (QTk . . . QT1 )A(Q1 . . . Qk ).

This procedure converges such that Ak is diagonal. The convergence rate is

given by
 k !
 λi 
[Ak ]ij = O   for i > j,
λj
and where the eigenvalues are sorted as

|λ1 | > |λ2 | > |λ3 | > ... > |λn | > 0.

Furthermore,

lim Qk = I, lim Ak = diag(λ1 , ..., λn ).

k→∞ k→∞

Define V = Q1 ...Qk . Then Λ = V T AV, and AV = V Λ, so that V is a matrix

of eigenvectors and Λ is a diagonal matrix of eigenvalues.
The complete QR algorithm is then:

1. Given a N × N matrix A, create N × N matrix V and set V = I.

Choose ε > 0 as desired for accuracy.

2. Calculate QR decomp as A = QR.

3. Update A to A = RQ.

60
 4. Set V = V Q.

5. Check magnitude of all off-diagonal elements of A. If all are < ε, then

stop. Otherwise, go to step 2.

There are a variety of improvements to this algorithm that we won’t

discuss.

3.1.3 Description of QR algorithm

Think of A as

and let
u0
u 0 = a0 , q0 = ,
||u0 ||
u2
u0 = a2 − (q0 ∙ a2 )q0 −(q1 ∙ a2 )q1 , q2 = ,
||u2 ||

each time, subtract out the projection of ai onto the orthonormal subspace.
The general formulas are
i−1
X ui
ui = a i − (qj ∙ ai )qj , qi = .
||ui ||
j=0

We can show that {qi } is an orthonormal basis. Then

a0 = ||u0 ||q0
a1 = ||u1 ||q1 + (q0 a1 )q0
a2 = ||u2 ||q2 + (q0 a2 )q0 + (q1 a2 )q1 .

We can rewrite these as below, where the left matrix is orthogonal and the
right one is upper triangular.

61
3.2 Lecture 12: Solving nonlinear equations
Many equations are not solvable analytically. For these, numerical tech-
niques can be helpful.

Relaxation method: consider x = 2 − e−x . This nonlinear equation is

not solvable by analytical methods. However, there is a simple numerical
method to solve it: just iterate the equation, i.e., guess an initial value
x = x0 , and plug it in:

x1 = 2 − e−x0
x2 = 2 − e−x1
..
.
xi+1 = 2 − e−xi

If we’re lucky, this procedure converges to a fixed point, which is a solution

of the equation. Bring up 1-iterate.py for an example of this method.
When it works, this is easy to program (similar method to logistic map HW
problem). Issues with it include:
1. Equation must in form x = f (x), but sometimes you can get it with a
2
rearrangement, e.g., log x + x2 − 1 = 0 ⇐⇒ x = e1−x .

2. Equation might have more than one solution, but the method con-
verges to only one of them. To get around this, change the initial
value.

62
 2
3. For some functions, it might not converge at all. E.g., x = e1−x , easy
to see that the solution is x = 1.

Bring up 2-iterate.py. A useful idea is to invent the function, if possible.

For instance,
2
x = e1−x ⇐⇒ log x = 1 − x2
p
⇐⇒ x = 1 − log x.

Bring up 3-iterate.py. In this case, the method converges (wish we had

the files...)

When does convergence happen for the relaxation method? Sup-

pose that f (x) has sufficient smoothness, and the equation is x = f (x), with
a solution for x = x0 . Then consider how relaxation method does when x is
close to x0 . Taylor expanding, the value x1 after an iteration is terms of the
previous value x0 is

x1 = f (x0 ) = f (x0 ) + (x0 − x0 )f 0 (x0 ) + ...

But we know that f (x0 ) = x0 , so x1 − x0 = (x0 − x0 )f 0 (x0 ) + .... If higher

order terms are small, then we can interpret the above as meaning that the
distance between the guess and the true solution shrinks or expands by a
factor of |f 0 (x0 )| < 1, or |f 0 (x0 )| > 1, respectively. So the relaxation method
converges if
|f 0 (x0 )| < 1.
2
So we can see why this method failed before. We took f (x) = e1−x ,
with x0 = 1, then
|f 0 (x0 ) = | − 2(1)(1)| = 2
and so does not converge.
On the other hand, if we invert the equation, we can see what happens:
go from x = f (x) to f −1 (x) = x. Define u = f −1 (x), then the desired
derivative is du/dx. But then we know that

dx
x = f (u) =⇒ = f 0 (u) = f 0 (f −1 (x))
du
which implies
du 1
= 0 −1 .
dx f (f (x))

63
But since f −1 (x0 ) = x0 , we have that

df −1 (x)  1
 = 0 0 .
dx x=x0 f (x )
So if |f 0 (x0 )| > 1, you can just invert everything, and be guaranteed conver-
gence!

When convergence isn’t guaranteed We can’t invert all equations

though. So sometimes we can’t guarantee convergence. Consider
x = x2 + sin x.
This has solution at x = 0. But applying the relaxation method gives
|f 0 (x0 )| = 2 > 1. There is the relation x = sin−1 (x − x2 ), but it’s not a true
inverse. It turns out though, we can use it to converge to the right solution.

3.2.1 Rate of convergence for relaxation method

It is exponentially fast if |f 0 (x0 )| < 1. But this isn’t very practical. How
accurate is the answer, exactly? And we want to be able to stop once our
desired accuracy is reached (e.g., if each iteration takes lots of time).
Let εi be ther error on the ith estimate, so that x0 = xi + εi , and x0 =
xi+1 + εi+1 . Neglecting higher order terms, from Taylor expansions we get
εi+1
εi+1 = εi f 0 (x0 ), and x0 = xi + εi = xi + 0 0
f (x )
Then using x0 = xi+1 + εi+1 , we have
εi+1
xi+1 + εi+1 = xi + 0 0
f (x )
 
1
=⇒ εi+1 1 − 0 0 = xi − xi+1
f (x )
xi − xi+1
=⇒ εi+1 = 1 ,
1 − f 0 (x 0)

then assuming xi ≈ x0 so that f 0 (xi ) ≈ f 0 (x0 ), we get

xi − xi+1
εi+1 = 1 .
1 − f 0 (x i)

So this is our estimated error, and we can keep repeating until this error
falls below some threshold.

64
What if we don’t know the derivative? Then just estimate it. Con-
sider three successive points, xi , xi+1 , xi+2 . From before, we have
xi+1 − xi+2
εi+2 =
1 − f 0 (x
1
0)

xi+1 − xi+2
≈ .
1 − f 0 (x
1
i)

Then approximate
f (xi ) − f (xi+1 )
f 0 (xi ) ≈ ,
xi − xi+1
and since xi+1 = f (xi ), and xi+2 = f (xi+1 ),
xi+1 − xi+2
=⇒ f 0 (xi ) ≈
xi − xi+1

So substituting back in, we get

xi+1 − xi+2 (xi+1 − xi+2 )2

εi+2 ≈ = .
1− xi −xi+1
xi+1 −xi+2
2xi+1 − xi − xi+2

So we can estimate the error even if we don’t know the derivative.

3.2.2 Relaxation method for 2 or more variables

Suppose we have N equations and N variables. Rewrite the implicit equa-
tions as

x1 = f1 (x1 , ..., xN )
..
.
xN = fN (x1 , ..., xN ).

Then choose an array of starting values and apply repeatedly. May or

may not converge. Recall the condition for single variable convergence was
|f 0 (x0 )| < 1. For the multivariable case, it is |λi | < 1 ∀i ∈ 1, ..., N , where λi
are eigenvalues of the Jacobian of f1 , ..., fN . Recall that this is the matrix
with entries ∂fi /∂xi , and so what we are saying is that need less eigenvalues
of magnitude less than 1.

65
3.2.3 Bisection method
This is what I’ve already seen in Math467. First, specify an interval [x1 , x2 ]
on which we would like a solution. Rearrange the equation so it has the
form f (x) = 0. Our goal is then to find the roots.
Calculate the values f (x1 ),f (x2 ). Then suppose f (x1 ) > 0, and f (x2 )<
0. If f (x) is continuous, there there exists one point between x1 and x2 such
that f is 0.

Also, if there is exactly one root in [x1 , x2 ], then f (x1 ) and f (x2 ) must have
the root to one side of the midpoint. Let x ˉ = x1 +x
2
2
be that midpoint. Eval-
uate f (ˉx). It could be that f (ˉ
x) = 0, in which case we’re done. Otherwise,
Then f (ˉ x) is either > 0 or < 0, and so has the same sign as either f (x1 )
or f (x2 ), and opposite sign to the other. If f (ˉ x) has opposite sign to f (x1 ),
then x1 and x ˉ bracket a root. So we’ve shortened the distance between x1
and x ˉ by a factor of two. Repeat the process.
The accuracy for this method improves exponentially. Suppose the initial
distance is Δ = x2 − x1 . The distance is halved at each step, so that after
N steps we get a distance Δ/2N . We stop the calculation when ε = Δ/2N ,
implying that N = log2 (Δ/ε ). For instance, suppose Δ = 10 10 and ε =
10−10 . Then N = log2 (1020 ) ≈ 67.
Disadvantages of the binary search / bisection technique include:

66
 1. Doesn’t work if f (x1 ) and f (x2 ) have the same sign (however, you
might know more about the function, and be able to shift it),

2. Can’t find even-order polynomial roots, such as for (1−x)2 or (2−3x)4 ,

3. Doesn’t extend to multiple dimensions.

3.2.4 Newton’s method

Again, we convert the problem of finding a solution to that of finding a root.
Start with a guess x, and then use the slope at x to make the next guess.

Then we use the derivative f 0 (x) to make the next guess x1 ,

f (x0 )
x1 = x0 − .
f 0 (x0 )
Do this, and keep iterating until convergence occurs.
How good is the accuracy? Suppose x0 is the value of the root. Then
Taylor expand about the estimate xi :
1
f (x0 ) = f (xi ) + (x0 − xi )f 0 (xi ) + (x0 − xi )2 f 00 (xi ) + ...
2
But x0 is such that f (x0 ) = 0. So the LHS vanishes. Divide by f 0 (x), and
rearrange as
 
0 f (xi ) 1 f 00 (xi )
x = xi − 0 − (x0 − xi )2 0 + ...
f (xi ) 2 f (xi )
| {z }
new estimate, xi +1

67
So then
1 f 00 (xi )
x0 = xi+1 − (x0 − xi )2 ,
2 f (xi )
and the ith error is given by

−1 f 00 (xi ) 2
εi+1 = ε
2 f 0 (xi ) i

Which means that Newton’s method has quadratic convergence. In other

f 00 (xi )
words, it converges very quickly, for certain values. If the term −12 f 0 (xi )
happens to be constant near the root, then the error after N iterations will
N
be roughly (cε0 )2 /c. This means doubly exponentially fast convergence.
However, there are disadvantages:

1. May not know the derivative (in which case, estiamte analytically)

2. Might not converge if f 0 (x) is very small - could give a larger error

3. In multiple dimensions, need to generalize to gradient and Hessian. To

estimate the error, the Hessian needs to be invertible (not always the
case). If we don’t know the gradient, need to estimate it.

3.3 Lecture 13: Secant method; generalized New-

ton’s; semidefinite programming
We saw above that we could use Newton’s method as a first-shot way to
solve nonlinear equations. The secant method is a modification of Newton’s
method. If we don’t know the derivative, just numerically estimate it.

Algorithm: begin with x1 , x2 . Now calculate

f (x2 ) − f (x1 )
f 0 (x2 ) ≈ .
x 2 − x1
Substitute the estimate back into Newton’s method equation:
x2 − x1
x3 = x2 − f (x2 )
f (x2 ) − f (x1 )

Convergence will be fast like Newton’s method, under good conditions.

68
3.3.1 Newton’s method for multiple variables
Suppose we have simultaneous nonlinear equations. We can write them as

f1 (x1 , ..., xN ) = 0
..
.
fN (x1 , ..., xN ) = 0

Note there should be the same number of functions as variables for the
problem to be tractable. Suppose there is a solution, q 0 = (x01 , ..., x0N ) such
that
fi (q 0 ) = 0 ∀i ∈ 1, ..., N
then Taylor expand about the solution:
X ∂fi
fi (q 0 ) = fi (x1 , ..., xN ) + (x0j − xj ) + ...
∂xj
j

We could also write this in vector notation as

− →
→ − →
− − −→ → − −
f ( q 0 ) = f (→
x ) + ∇f ∙ ( x0 − →
x ) + ...

We’ll continue with vector notation, but with the arrows implicit. Note here
that ∇f is the Jacobian matrix with entries ∂fi /∂xj . Since x0 was assumed
to be a solution, this implies that f (x0 ) = 0.

=⇒ 0 = f (x) + ∇f ∙ (x0 − x)
⇐⇒ ∇f ∙ (x − x0 ) = f (x).

If we set Δx = x − x0 , then

⇐⇒ ∇f ∙ (Δx) = f (x).

This is a matrix equation, so we can solve it by Gaussian elimination. After

doing so, we get Δx and then the next estimate is

x0 = x − Δx.

If we can’t get the Jacobian matrix, then we can numerically estimate

it. This is basically generalizing the vector secant method.

69
Maxima and minima of functions
Finding optima is clearly closely related to root finding. To introduce op-
timization, we’ll stick to optimziation of a single function, i.e., a minimum
of f (x1 , ..., xN ). Functions can have more than one minimum or maximum.
There is also a distinction between local and global optima.
The standard method for computing minima/maxima is to solve
∂f
= 0 ∀i.
∂xi
If the equations are linear, then we can use Gaussian elimination for this.
If they are nonlinear, use Newton’s method. Many times though, we can’t
calculate the derivative, so we’ll need alternate techniques.

Golden ratio technique

Similar to binary search. This works for finding the minima of single-variable
functions. It also doesn’t distinguish between local and global minima. Sup-
pose our picture is

First we pick two points, x1 and x4 which correspond to an interval where

we would like to search for a minimum. Then pick x2 and x3 such that
x1 < x2 < x3 < x4 (we’ll give their spacing later). Suppose that

f (x2 ) < min{f (x1 ), f (x4 )}, or

f (x3 ) < min{f (x1 ), f (x4 )}

70
Then we know that the minimum lies between them. Now compare f (x2 )
with f (x3 ). If f (x2 ) < f (x3 ), then the picture is

and the minimum is between x1 and x3 , and otherwise, the picture is

and the minimum is between x2 and x4 . We can keep repeating this process
by adding another point to the three that we narrow down to.
Now, the obvious question: how do we pick x1, x2 , x3 , x4 ? Well, x1 and
x4 are fixed. It makes no sense to favor one side over the other, so x2 and

71
x3 should be symmetric about the midpoint. So pick
 
x1 + x 4 x 1 + x4
− x2 = x 3 − ,
2 2

which means x2 − x1 = x4 − x3 . Then we need another equation to pin down

the next location for x2 and x3 . For one iteration, if f (x2 ) < f (x3 ), then
we would choose the next point between x1 and x3 . The issue is we want to
figure out the “optimum space” to shave off the interval at each new point.
Do this by choosing the proportions of intervals on each iteration to be the
same. Supposing that we are always “going left”, this gives the ratio φi of
interval sizes to be
x4 − x 1 x 2 − x 1 + x 3 − x1 x 2 − x1
φi = = = + 1.
x3 − x 1 x3 − x 1 x 3 − x1
For the next step, the ratio is
x3 − x 1
φi+1 = .
x2 − x 1
To make things fair for each iteration, we set the values equal, giving
1
φi+1 = φi = +1
φi+1
⇐⇒ φ2i+1 − φi+1 − 1 = 0.

Solving this gives the golden ratio:

√
1+ 5
φi+1 = .
2
So the complete algorithm for the golden ratio method is:

1. Pick two initial points about the minimum, x1 and x4 . Calculate x2

and x3 according to the golden ratio rule. Calculate f (x) fat all four
points.

2. If f (x2 ) < f (x3 ), then x4 = x3 , x3 = x2 . Pick a new x2 according to

the golden ratio, and calculate f (x2 ).

3. Else, set x1 = x2 , x2 = x3 , and get x3 and f (x3 ).

4. If |x4 − x1 | > 0, go to 2. Else, calculate (x2 + x3 )/2.

72
Example: Buckingham potential An approximate representation of
the potential energy of interaction between atoms in a solid or gas, as a
function of the distance r between them.
  
σ 6
V (r) = V0 − e−r/σ .
r

There are two terms: a short-range repulsive force, and a longer-range at-
tractive force. There’s no known analytic expression for the ‘resting dis-
tance’.

Bring up the file buckingham.py. The golden ratio search cannot be gen-
eralized to functions of more than one variable. Instead, we can use the
Gauss-Newton method.
For this, to find a minimum of f (x), set f 0 (x) = 0 and solve for the roots.
Using Newton’s method, we get the update rule

f 0 (xi )
xi+1 = xi − .
f 00 (xi )

This has fast convergence, and can be generalized to more than one variable.
If we can’t calculate the second derivative, we can approximate it by xi+1 =
xi − γf 0 (xi ), where γ is a constant value representing a guess for f 00 (x). This
is called the gradient descent method.

73
 We’re measuring a gradient at x, and subtracting a constant times the
gradient. For γ > 0 we converge to a minimum, and for γ < 0, we converge
to a maximum. The magnitude of γ controls the rate of convergence. If γ is
large, we go faster, but could ‘overshoot’ the solution. If we can’t calculate
the derivative, then just numerically estimate it, using two successive points.

3.3.2 Semidefinite programming

Semidefinite programming is an analytical and numerical tool for solving
optimization problems with semidefinite constraints. It can be useful in
e.g., finding the ground state of a Hamiltonian – it’s frequently used in
quantum information science for this purpose.
Let X be a Hermitian matrix (X † = X ) acting on a finite-dimensional
complex vector space, Ξ, and let Y be the same, on Θ. Let Φ denote a
linear ‘super-operator ’ which takes X ∈ Ξ to Y ∈ Θ, i.e., Φ(X) = Y .
Then Φ is Hermiticity preserving if Φ(X) ∈ Herm(Θ) for all X ∈
Herm(Ξ). Then a semidefinite program is a triple (Φ, A, B) where
1. Φ is Hermiticity preserving

2. A ∈ Herm(Ξ), and B ∈ Herm(Θ)

Then let hC, Di = Tr(C † D) be the Hilbert-Schmidt inner product between
the matrices C and D. Define the adjoint Φ† of a super-operator by

hE, Φ(F ) i = hΦ† (E), F i

which holds for all E ∈ L(Θ), and F ∈ L(Ξ). Associated with these Φ, A, B
are two optimization problems, called the primal and dual.
Primal:

max hA, Xi,

subject to Φ(X) =B, X ∈ Pos(Ξ).

while dual:

min hB, Y i,
subject to Φ† (Y ) ≥A, Y ∈ Herm(Θ).

In this case, hA, Xi is the objective (cost) function. Φ(X) = B and X ∈

Pos(Ξ) are constraints (note Pos is the set of positive semidefinite operators
acting on a set, in this case Ξ . So for X ∈ Ξ, xXx ≥ 0 for any x ). Primal
and dual problems have a special relationship.

74
 An operator X ∈ Pos(Ξ) for which Φ(x) = B is primal feasible. Similarly,
any Y ∈ Herm(Θ) that satisfies Φ† (Y ) ≥ A is dual feasible.
Let A = {X ∈ Pos(Ξ) : Φ(X) = B}. Similarly, let B = {Y ∈ Herm(Θ) :
Φ (Y ) ≥ A}. The primal optimum is
†

α = sup hA, xi
x∈A

and the dual optimum is

β = sup hB, Y i
y∈B
In this case, α and β might be either finite or infinite. If A = ∅ then
a = −∞, and if B = ∅ then β = ∞ .
If x ∈ A satisfies hA, xi = α, then we say x is the optimal primal solution.
Similarly, if Y ∈ B satisfies hA, xi = β then Y is the optimal dual solution.

Example: take Φ(X) = Tr(X), and B = 1. Then the primal problem is

max hA, Xi,
subject to Tr(X) = 1, X ∈ Pos(Ξ).
while the dual is
min Y,
subject to Y I ≥A, Y ∈ R.
Nifty. Semidefinite programming winds up having a powerful duality theory.
Stated as the weak duality: for every semidefinite program (Φ, A, B) it holds
that α ≤ β.
Proof: trivial if A = ∅ or B = ∅. For all X ∈ A and Y ∈ B, we have
hA, Xi ≤ hΦ† (Y ), Xi = hY, Φ(X)i = hY, Bi =⇒ α ≤ β.

3.4 Lecture 14: more semidefinite programming

This lecture is going to be very paraphrased, because I’ve seen this material
and frankly it’s pretty damn tedious. I’ll keep it brief.
Key results: every semidefinite program obeys weak duality, which is
the statement that α ≤ β. Strong duality is when α = β, and it turns out
it is precisely in this case that semidefinite programming can be a useful
analytical tool. One set of conditions to check for if it holds is the Slater
conditions. Those are the only things from this lecture that I’m going to
write about. (C’mon, the next one is on Fourier transforms!)

75
3.5 Lecture 15: Fourier series and transforms
Fourier transforms are useful for signal analysis, and are also an important
tool for solving differential equations. First let’s recall what Fourier series
can do: any periodic function f (x) defined on a finite interval 0 ≤ x ≤ L
can be written as a Fourier series.
If f (x) is symmetric about the midpoint at L/2, then we can write
∞
X  
2πkx
f (x) = αk cos ,
L
k=0

where {αk } is the set of coefficients. If f (x) is antisymmetric about the

midpoint, then we have
∞
X  
2πkx
f (x) = βk sin
L
k=1

So we can write a function with no symmetry as

∞
X   ∞
X  
2πkx 2πkx
f (x) = αk cos + βk sin
L L
k=0 k=1

and then making using of cos θ = (eiθ + e−iθ )/2 and sin θ = (eiθ − eiθ )/2i to
write
X∞     
2πkx 2πkx
f (x) = αk exp i + exp −i
L L
k=0
∞     
iX 2πkx 2πkx
+ βk exp −i − exp i
2 L L
k=1

From which point we can collect terms and write it as

∞
X  
2πkx
f (x) = γk exp i
L
k=−∞

where 
 2 (α−k + iβ−k ), k < 0
1

γ k = α0 , k=0

1
2 (α−k − iβ−k ), k > 0.

76
 Fourier series can only be used for periodic functions! To extend to non-
periodic ones, just pick out an interval of a function and repeat it infinitely
so that it becomes periodic.
2πkx
How do we calculate the coefficients, γk ? Just use the fact that {e−i L }k
constitutes an orthonormal basis for the space [0, L]. That is, consider that
Z L   X∞ Z L  
2πkx 2π(k 0 − k)x
f (x) exp −i = γk 0 exp i dx.
0 L 0 0 L
k =−∞

If k 0 6= k, then
Z L     
2π(k 0 − k)x L 2π(k 0 − k)x L
exp i dx = exp i
0 L i2π(k 0 − k) L 0
L  
= exp(i2π(k 0 − k)) − 1
i2π(k 0 − k)
= 0 since e2πin = 1 ∀n ∈ Z

However, if k 0 = k, then the integral is equal to L. In this case,

Z L  
2πkx
f (x) exp −i dx = Lγk
0 L
or Z  
1 L
2πkx
γk = f (x) exp −i dx.
L 0 L

3.5.1 Discrete Fourier transforms

There are many cases in which it isn’t possible to calculate the coefficients
γk analytically. So we can use numerical methods. It turns out that ap-
proximations with the trapezoidal rule is equivalent to the discrete Fourier
transform.
Consider N slices of width h = L/N. Applying the trapezoidal rule gives
 " N
X −1  #
1 L 1 1 2πkxn
γk = f (0) + f (L) + f (xn ) exp −i
L N 2 2 L
n=1

when the sample point positions are xn = nL/N. Since f (x) is periodic, we
have f (0) = f (L), so then above simplifies to
N −1  
1 X 2πkxn
γk = f (xn ) exp −i
N L
n=0

77
We can use this to evaluate coefficients, at least in cases with evenly sampled
data (pretty frequent). It’s also worth noting that while these results were
derived using the trapezoidal rule, there is a sense in which that are exact.
Recall that
N
X −1
1 − aN
ak = , a 6= 1,
1−a
k=0

then
N
X −1  k
2πm 1 − ei2πm
ei N = = 0,
k=0
1 − ei2πm/N
since m is an integer, making the numerator zero. In the case that m = 0,
or is a multiple of N , then the sum is N. So
N
X −1   (
2πkm N if m = 0, N, 2N, ...
exp i =
k=0
N 0 else

Then consider the sum

N −1   N −1
"N −1 #  
X 2πkn X X 2πkxn0 2πkn
ck exp i = γn0 exp −i exp i
N L N
k=0 k=0 n0 =0
N
X −1 N
X −1  
n − n0
= γ n0 exp i2πk( )
0
N
n =0 k=0
N
X −1
= yn0 δn,n0 N
n0 =0
= N yn
N −1  
1 X 2πkn
=⇒ yn = ck exp i .
N N
k=0

This is the inverse discrete Fourier transform (inv. DFT).

This proves that the matrix with entries
 
1 2πkn
Ukn = √ exp −i
N N

is a unitary matrix. So we can recover the original values exactly by per-

forming the inverse DFT. So you can move freely back and force between
the original values and the Fourier coefficients.

78
 • We can compute this on a computer because the sum is finite
• This discrete formula only gives sample values yn = f (xn ). So if the
function is oscillating rapidly between samples, the DFT won’t capture
this, so DFT just gives some idea of the function.
If the function is real, then can use this symmetry to simplify further. Sup-
pose all yn are real and consider ck for N/2 < k ≤ N − 1, so k = N − r for
1 ≤ r < N/2. Then
N
X −1  
2π(N − r)n
cN −r = yn exp −i
N
n=0
N
X −1  
2πrn
= yn exp (−i2πn ) exp i
N
n=0
N
X −1  
2πrn
= yn exp i = c0r
N
n=0

so then cN −1 = c10 , cN −2 = c02 , etc. So when calculating the DFT of a real

function, we only have to calculate ck for 0 ≤ k < N/2. However, if the yn
are complex, then we need to calculate all N Fourier coefficients.
Bring up dft.py. This program uses exp from the cmath package, which
isn’t the quickest way to calculate the DFT. We can instead do FFT. If we
shift the positions of the sample points. then not much changes. Suppose
that instead of taking samples at xn = nL/N, we take them at x0n = xn + Δ.
Then
N
X −1  
2πk(xn + Δ)
ck = f (xn + Δ) exp −i
L
n=0
  N −1  
2πkΔ X 0 2πkxn
= exp −i f (xn ) exp −i
L L
n=0
  NX−1  
2πkΔ 0 2πkxn
= exp −i yn exp −i ,
L L
n=0

where yn0 = f (x0n )are the new samples.

We can absorb Pthe phase factors into

N −1 0
the coefficients as c0k = exp i 2πkΔ
L c k so that c 0 =
k n=0 yn exp −i L
2πkn

so that DFT is independent of where the samples are taken.

We can distinguish between Type-I DFT where we divide interval [0, L]
into N slices and take samples at endpoints, and a Type-II where we take
samples at the midpoints of slices.

79
3.5.2 2D Fourier transform
It’s useful for image processing, for instance in astronomy (classic case: Hub-
ble image correction). Suppose we have M × N grid of samples ymn . First
do a FT on the rows:
N
X −1  
0 2πln
cml = ymn exp −i ,
N
n=0
and then FT the m variable:
NX −1  
0 2πkm
ckl = cml exp −i .
M
n=0
Combined, these read
N
X −1 N
X −1   
km ln
ckl = ymn exp −i2π + .
M N
m=0 n=0
What is the FT doing? Breaking down a signal into its frequency com-
ponents, like a signal analyzer. Bring up dft.py. The first spike is the
frequency of the main wave, and the others are harmonics.

Discrete cosine transform

Recall that if a function is symmetric about x = L/2 (the midpoint) then
we can write  
∞
X 2πkx
f (x) = αk cos
L
k=0
We cannot do this for all functions. However, if we’d like to do so, we can
by just sample a function over an interval, and then adding it to its mirror
image, i.e.,

80
So we make the function symmetric, and when the samples are, we have
y0 = yN , y1 = yN −1 , y2 = yN −2 , etc. We then get for the DFT:
N
X −1  
2πkn
ck = yn exp −i
N
n=0
N/2
X   N
X −1  
2πkn 2πkn
= yn exp −i + yn exp −i
N N
n=0 n=N/2+1
N/2
X   N
X  
2πkn 2πk(N − n)
= yn exp −i + yN −n exp i
N N
n=0 n=N/2+1

where in the final line we used exp(i2πk) = 1. Make a change of variables

N − n → n to get
N/2
X   N/2−1
X  
2πkn 2πkn
ck = yn exp −i + yn exp i
N N
n=0 n=1
  N/2−1
X  
2πk(N/2) 2πkn
= y0 + yN/2 cos +2 yn cos .
N N
n=1

Usually though, the discrete cosine transform is applied to real values, which
means that the ck coefficients are real. In this case, we have the cN −r =
c0r = cr , and the inverse transform is
N −1  
1 X 2πkn
yn = ck exp i
N N
k=0
 
N/2   N −1  
1 X 2πkn X 2πkn 
= ck exp i + ck exp i
N N N
k=0 k=N/2+1
 
N/2
X   N
X −1  
1  2πkn 2π(N − k)n 
= ck exp i + cN −k exp −i
N N N
k=0 k=N/2+1
 
N/2   N/2−1  
1 X 2πkn X 2πkn 
= ck exp i + ck exp −i
N N N
k=0 k=1
 
  N/2−1
X  
1  2πn(N/2) 2πkn 
= c0 + cN/2 cos +2 ck cos
N N N
k=1

81
which is the inverse discrete cosine transform. It has so much symmetry that
the DCT is the same as its inverse! If we take the samples at midpoints,
then we can show that the coefficients are
N/2−1
X  
2πk(n + 1/2)
ak = 2 yn cos ,
N
n=0

and the inverse transform is

 
N/2−1
X  
1  2πk(n + 1/2) 
yn = a0 + 2 ak cos .
N N
k=1

A nice feature of the DCT is that it does not assume that the function
is periodic. Neither does the DFT, but it does force the first and last values
to be the same, which can create a large discontinuity. The DCT does not
do this. N.b. we can also calculate the discrete sine transform, but this is
rarely used because it forces the endpoints to zero.

82
Chapter 4

Lectures 16-20

We’ve gotten pretty far in the lecture notes (3/4 complete), and a little over
halfway in the homeworks. In all honestly, I’m feeling a bit burned out on
the homeworks; likely because of both having written crap code on the first
half of HW3 without internet access, and jamming through the lectures in
cars and planes without taking time to focus on the content. But this means
a couple things: (1) I’m comfortable enough writing code in Python to do
non-trivial things without constantly checking StackExchange, and (2) I’ve
gotten pretty damn far. We’re going to keep pushing, with more of a concept
focus. I’m still committing to doing the good problems from homeworks 3
and 4, and most of the remaining lectures look pretty interesting. I can put
in 3hours/day on this to wrap it up before my semester starts.
I’m at a reasonable level with BaKoMa TEX too. It’s obvious that some
functionalities – especially matrix operations, or even long lines of math
derivations, will be much easier handwritten. This is fine, with figure input.

4.1 Lecture 16: Fast Fourier transform, ordinary

diffeqs, Euler method, Runge-Kutta
Recall that the discrete Fourier transform is
N
X −1  
2πkn
ck = γn exp −i .
N
n=0

Fourier analysis just refers to analyzing the distinct components that

contribute to a periodic phenomena. In other words, it’s about expression a
function as a sum of periodic components, and then recovering those com-
ponents. A simple example would be to consider a single-frequency whistle.

83
An audio detector that senses compressions and rarefractions of air would
produce a sinusoidal voltage when this whistle is blown. Taking the DFT
of that signal would yield a single-frequency peak in the whistle’s frequency
spectrum. This gets useful when combining many periodic signals together.
The Python program (available to the people taking the class) had a for-
loop for each coefficient, and N terms in the sum, implying N 2 operations
would be required to get all the coefficients. If we’re not willing to wait for
more than 1 billion operations, then we can do a DFT for N 2 = 109 =⇒
N ≈ 32000 samples. This isn’t too much - about one second of audio.
The fast Fourier transform (FFT) is a DFT-solving algorithm for cutting
the number of computations needed for N points from 2N 2 to 2N ln n. It
was discovered by Cooley and Tukey in 1965, although Gauss had pointed
at the key step in the algorithm in 1805.
The algorithm is easiest to describe when the number of samples is a
power of two. Let N = 2m , for m an integer. Consider the sum in the DFT
equation, and divide it into 2 groups of even and odd terms. Consider the
even terms first, where n = 2r, for r ∈ {0, ..., N/2 − 1}. Then
N/2−1
X  
2πk(2r)
Ek = y2r exp −i
N
r=0
N/2−1
X  
2πkr
= y2r exp −i ≡ Ek,modN/2 .
N/2
r=0

Note that this is just another DFT, with N/2 samples instead of N . Then
look at the odd terms:
N/2−1
X   N/2−1
X  
2πk(2r + 1) −i2πk/N 2πkr
y2r+1 exp −i =e y2r+1 exp −i
N N/2
r=0 r=0
= e−i2πk/N Ok

where Ok is another DFT with N/2 sample points. Call this Ok,modN/2 . So
then

ck = Ek,modN/2 + e−i2πk/N Ok,modN/2

= Ek,modN/2 + Wnk Ok,modN/2 .

We can summarize what we have in terms of a diagram. Let

84
denote a N -point DFT. The output are the DFT of the input. Then to
compute c0 , we do

Figure 4.1: “Butterfly” diagram for FFT.

This diagram specifies that recursion. The starting point is DF T1 which

is just the identity transformation. Then at the mth stage that are O(N )
calculations. There are ? ln N stages, meaning that we have O(N ln N )
operations. In other words, it’s nearly linear!
For example, if we have N = 106 samples to process, then the naive
way would require 1012 operations. This is highly nontrivial for a typical
computer, and not practical. But N ln N = 107 , since the natural logarithm
of anything big is like, 10. This can then be done in under a second! The
inverse DFT can be done in the same way.
Numpy provides numpy.fft as a FFT package. Skimming the documen-
tation, you’ll find there are different types of FFTs within it. Rfft computes

85
the FFT for a set of real numbers, and returns the first half (since the
other half are complex conjugates). You can also use fft and ifft to calcu-
late Foruier transforms of complex data. There are also functions for two
dimensional transforms, and well as functions for higher dimensions.

4.1.1 Ch. 8: Solving Ordinary Differential Equations

Consider the first-order equation

dx 2x 3x2
= + 3 .
dt t t
It is not separable, and it’s also nonlinear, so we need to use computational
methods to solve it.
The general form for a first-order differential equation is
dx
= f (x, t).
dt
To calculate a full solution, we need a boundary condition, e.g., the value of
x at one particular value of t (usually t = 0).

Euler’s method
Suppose we have to solve dx
dt = f (x, t), and we’re given an initial condition.
Then Taylor expand x(t + h) about t, to get

h2 00
x(t + h) = x(t) + hx0 (t) + x (t) + ...
2
= x(t) + hf (x, t) + O(h2 ),

so if we neglect O(h2 ) terms, we get

x(t + h) = x(t) + hf (x, t).

So if we know x at time t, we can just use this equation to iterate. If h is

small enough, this does pretty well. It’s called Euler’s method. There’s an
example in 2-euler.py, for dx/dt = −x3 + sin t, but it’s not online /. So
we write our own:
#Numerical a n a l y s i s o f dx/ dt = −x ∗∗3 + s i n ( t )
import numpy a s np
import m a t p l o t l i b . p y p l o t a s p l t

def f (x , t ) :

86
 r e t u r n −x ∗∗3 + np . s i n ( t )

x0 = 0 .
xMax = 8 0 .
h = (xMax − x0 ) /1 e5
t = np . a r a n g e ( x0 , xMax , h )
x = np . z e r o s ( np . s i z e ( t ) )

f o r i i n r a n g e ( np . s i z e ( t ) ) :
i f i == 0 :
x [ i ] = x0
else :
x [ i ] = x [ i −1] + h∗ f ( x [ i −1] , t [ i −1])

plt . plot (t , x)
p l t . show ( )

which yields

Runge-Kutta method
In reality, no one uses Euler’s method, since Runge-Kutta is a much-more
accurate approach that’s of comparable difficulty to program. The error in

87
 2
one step of the Euler method is to leading order 12 h2 ddt2x . There error for
many steps from t = a to t = b using step size h is then N = (b − a)/h. The
value of t for steps are at tk = a + kh, and xk is the corresponding value of
x. Then the total error is
N −1   "N −1   #
X 1 2 d2 x 1 X df
h = h h
2 dt2 x=xk 2 dt x=xk ,t=tk
k=0 k=0
Z b
1 df
≈ h dt
2 a dt
1
= h [f (x(b), b) − f (x(a), a)] .
2
So the approximation is good if h is small. The error is linear in h, so that
if we make it smaller, the error is smaller. The Runge-Kutta method can
do much better (n.b., it’s really a set of methods).
Euler’s method can be expressed graphically as

where the curve is x(t), and the diffeq dx/dt = f (x, t) says that the slope
of the solution is equal to f (x, t). So the Euler method extrapolates to time
t + h, giving an estimate of x(t + h).
Second order Runge-Kutta method is to use the slope at the midpoint,
t + h/2, to extrapolate to t + h. Perform a Taylor expansion of x(t + h) about
t + 12 h :

88
while x(t) has the expansion (still about t = t + h/2):

then subtracting the second from the first, we get

 
df
x(t + h) = x(t) + h + O(h3 )
dt t+h/2
= x(t) + hf (x(t + h/2), t + h/2) + O(h3 ).

This means that the O(h2 ) term has disappeared! The error term is now
O(h3 ).
The problem though, is that this approach requires an estimate of x(t +
h/2), which we don’t have (we only know x(t)). Instead, approximate x(t +
h/2) using Euler’s method:
1
x(t + h/2) ≈ x(t) + hf (x, t),
2
and then substitute. So we get

k1 = hf (x, t)
1 1
k2 = hf (x + k1 , t + h)
2 2
so that
x(t + h) = x(t) + k2
is the final Runge-Kutta estimate, to second order.
How can we be sure that our error of h3 is still intact, even having
estimated x(t + h/2) with Euler’s method? To do this, consider the Taylor

89
expansion of f (x + k1 /2,t+h/2) around x(t + h/2) :
   
1 1 1 1
f x(t) + k1 , t + h =f x(t + h), t + h +
2 2 2 2
  
1 1 ∂f
x(t) + k1 − x(t + h) +
2 2 ∂x x(t+h/2),t+h/2
 2 !
1 1
O x(t) + k1 − x(t + h) .
2 2

From before we have that

 
1 1

x t + h = x(t) + hf (x, t) + O h2
2 2
1

= x(t) + k1 + O h2
2
which implies
 
1 1
1 1
x(t) + k1 − x(t + h) = O h + f x(t) + k1 , t + h
2
2 2 2 2
 
1 1 1

= f x(t + h) + k1 , t + h + O h2
2 2 2
 
1 1

=⇒ k2 = hf x(t + h), t + h + O h3 .
2 2

So indeed, the error introduced by the Euler approximation is order h3 . The

ideal thing now would be to see 3-rk2.py, but I guess this isn’t happening.
From this point, you can push the Runge-Kutta method further. Take
more Taylor expansions, and cancel out higher order terms. The most pop-
ular method is the fourth order Runge-Kutta method, which is still
relatively simple to program:

k1 = hf (x, t)
 
1 1
k2 = hf x + k1 , t + h
2 2
 
1 1
k3 = hf x + k2 , t + h
2 2
k4 = hf (x + k3 , t + h) ,
1
x(t + h) = x(t) + (k1 + 2k2 + 2k3 + k4 ) .
6

90
This is accurate to terms of order h4 , and has error of order h5 . The deriva-
tion is complicated, but the final equations are simple. For the added coding
complexity of five equations, we get three orders of h more accurate than
Euler’s method. An implementation would look like:
#dx/ dt = −x ∗∗3 + s i n ( t ) w/ Runge−Kutta ( 4 th o r d e r )
#same preamble a s E u l e r method ( s e e above )

f o r i i n r a n g e ( np . s i z e ( t ) ) :
i f i == 0 :
x [ i ] = x0
else :
k1 = h∗ f ( x [ i −1] , t [ i −1])
k2 = h∗ f ( x [ i −1]+k1 / 2 , t [ i −1]+h / 2 )
k3 = h∗ f ( x [ i −1]+k2 / 2 , t [ i −1]+h / 2 )
k4 = h∗ f ( x [ i −1]+k3 , t [ i −1]+h )
x [ i ] = x [ i −1] + ( 1 / 6 ) ∗ ( k1 + 2∗ k2 + 2∗ k3 + k4 )
plt . plot (t , x)

91
Finding solutions over infinite ranges
Sometimes, we want to go out to t = ∞. The idea to do this, as before, is
change of variables. Let u = t/(1 + t), or t = u/(1 − u). Then u → 1 as
t → ∞. Rewrite the diffeq with the chain rule as
dx du
= f (x, t), or
du dt  
dx dt u
= f x, .
du du 1−u
But 
dt 1 u
= f x,
du (1 − u)2 1−u
so  
dx 1 u
= f x, .
du (1 − u)2 1−u
Then let
 
u
g(x, u) = (1 − u) f 2
x, , so
1−u
dx
= g(x, u),
du
and then just use techniques from before.

Example Suppose we want to solve

dx 1
= 2
dt x + t2
from t = 0 to t = ∞. We are given x = 1 at t = 0. Then instead, solve
1 1
g(x, u) =
(1 − u) x +
2 2 u2
(1−u)2
1
= ,
x2 (1 − u)2 + u2
so from u = 0 to u = 1, solve
dx 1
= 2 .
du x (1 − u)2 + u2
Check out 5-odeinf.py to see it in action.

92
4.2 Lecture 17: Simultaneous ordinary diffeqs, adap-
tive step size Runge-Kutta, leap frog method.
Let’s consider differential equations with more than one variable. Suppose,
for instance, that we want to solve
dx dy
= xy − x, = y − xy + sin2 ωt.
dt dt
These are still ordinary differential equations. The more general form would
be
dx dy
= fx (x, y, t), = fy (x, y, t).
dt dt
If there are more variables, write this as
dr
= f (r, t),
dt
where r is a vector of functions. Keeping in the theme of last lecture, we
can Taylor expand r(t) about t = t + h :
 
dr

r(t + h) = r(t) + h + O h2
dt t=t


= r(t) + hf (r, t) + O h2 .

Dropping the order h2 term gives the vector Euler method:

r(t + h) = r(t) + hf (r, t)

and we can similarly generalize the Taylor expansions for the Runge-Kutta
method to get the vector case:

k1 = hf (r, t)
 
1 1
k2 = hf r + k1 , t + h
2 2
 
1 1
k3 = hf r + k2 , t + h
2 2
k4 = hf (r + k3 , t + h) ,
1
r(t + h) = r(t) + ( k1 + 2k2 + 2k3 + k4 ) ,
6
which can be viewed in 1-odesim.py.

93
4.2.1 Second order and higher diffeqs
Obviously, not all equations have only first derivatives. We can solve second
order equations using a simple trick to reduce them to first order. The
general form for a second order diffeq with one dependent variable is
 
d2 x dx
= f x, , t ,
dt2 dt
where f is an arbitrary function. For instance,
 
d2 x 1 dx 2 dx
= =2 − x3 e−4t
dt2 x dt dt

Define y = dt .
dx
Then we can write the above as
dy
= f (x, y, t).
dt
Then these two equations are equivalent to the first. In other words, we’ve
reduced a single 2nd order diffeq to two 1st order ones. We can do the same
trick for higher order too. For example,
 
d3 x dx d2 x
= f x, , 2 , t .
dt3 dt dt
dy
Set y = dt ,
dx
and z = dt . Then we have
dz
= f (x, y, z, t).
dt
We can generalize this to the vector case too. Suppose that we have
 
d2 r dr
= f r, , t .
dt2 dt
This is equivalent to
dr ds
= s, = f (r, s, t).
dt dt
So if we have n equations of mth order, then this method gives n × m
simultaneous first order equations.
For example, consider a nonlinear pendulum. The standard treatment
here would be to make this linear, but we can treat it in full generality,
numerically.

94
 2
The acceleration of the mass in the tangential direction is ` ddt2θ , and the force
on the mass is vertically downward, with magnitude mg. The component of
the force in the tangential direction is thus mg sin θ. Then Newton’s second
law gives the equation of motion,

d2 θ
m` = −mg sin θ
dt2
d2 θ
⇐⇒ ` 2 = −g sin θ
dt
which you can solve numerically using
dθ dω g sin θ
= ω, =− .
dt dt `
This is visible in 2-pendulum.py, which I can’t access. Writing my own shit
instead, we see
import numpy a s np
import m a t p l o t l i b . p y p l o t a s p l t

d e f f T h e t a ( omega ) :
r e t u r n omega
d e f fOmega ( t h e t a , g , l ) :
r e t u r n g ∗np . s i n ( t h e t a ) / l ∗( −1)

t h e t a 0 = np . p i / ( 2 ∗ 1 0 0 )
omega0 = 0 .
t0 = 0 .
tMax = 2 0 .
h = ( tMax − t 0 ) /1 e4

95
t = np . a r a n g e ( t0 , tMax , h )
t h e t a = np . z e r o s ( np . s i z e ( t ) )
omega = np . copy ( t h e t a )
g , l = 9.81 , 0.5

f o r i i n r a n g e ( np . s i z e ( t ) ) :
i f i == 0 :
t h e t a [ i ] , omega [ i ] = t h e t a 0 , omega0
else :
t h e t a [ i ] = t h e t a [ i −1] + h∗ fTheta ( omega [ i −1])
omega [ i ] = omega [ i −1] + h∗fOmega ( t h e t a [ i −1] , g , l )

plt . x l a b e l ( ’ $t$ ’ )
plt . y l a b e l ( ’ $\ theta$ ’ )
plt . t i t l e ( ’ Pendulum v i a E u l e r method ’ )
plt . plot ( t , theta )
plt . show ( )

Now: why would the angle be getting bigger? My bet would be propagation
of some numerical error. It turns out, if we program the same thing but
with Runge-Kutta we get a similar error (but we’ll see a method later that
fixes this):
import numpy a s np
import m a t p l o t l i b . p y p l o t a s p l t

96
d e f f T h e t a ( omega ) :
r e t u r n omega
d e f fOmega ( t h e t a , g , l ) :
r e t u r n g ∗np . s i n ( t h e t a ) / l ∗( −1)

t h e t a 0 = np . p i / ( 2 ∗ 1 0 0 )
omega0 = 0 .
t0 = 0 .
tMax = 2 0 .
h = ( tMax − t 0 ) /1 e4
t = np . a r a n g e ( t0 , tMax , h )
t h e t a = np . z e r o s ( np . s i z e ( t ) )
omega = np . copy ( t h e t a )
g , l = 9.81 , 0.5

f o r i i n r a n g e ( np . s i z e ( t ) ) :
i f i == 0 :
t h e t a [ i ] , omega [ i ] = t h e t a 0 , omega0
else :
k1 = h∗ f T h e t a ( omega [ i −1])
k2 = h∗ f T h e t a ( omega [ i −1]+k1 / 2 )
k3 = h∗ f T h e t a ( omega [ i −1]+k2 / 2 )
k4 = h∗ f T h e t a ( omega [ i −1]+k3 )
t h e t a [ i ] = t h e t a [ i −1] + ( 1 / 6 ) ∗ ( k1 + 2∗ k2 + 2∗ k3 + k4 )

K1 = h∗fOmega ( t h e t a [ i −1] , g , l )
K2 = h∗fOmega ( t h e t a [ i −1]+K1/ 2 , g , l )
K3 = h∗fOmega ( t h e t a [ i −1]+K2/ 2 , g , l )
K4 = h∗fOmega ( t h e t a [ i −1]+K3 , g , l )
omega [ i ] = omega [ i −1] + ( 1 / 6 ) ∗ (K1 + 2∗K2 + 2∗K3 + K4)

plt . x l a b e l ( ’ $t$ ’ , f o n t s i z e = 20)

plt . y l a b e l ( ’ Theta ’ , f o n t s i z e = 2 0 )
plt . t i t l e ( ’ Pendulum v i a Runge−Kutta ’ , f o n t s i z e = 2 0 )
plt . plot ( t , theta )
plt . show ( )

97
Varying step size when solving diffeqs
So far, we’ve only looked at using repeated steps of the same size. We can
do better by varying the step size. It’s worthwhile to analyze this. Suppose
our function looks like

Then in the slowly varying regions, we’d like to take larger step sizes, but
in the quickly varying regions, we should take smaller step sizes. So we
want to vary the step size such that the error introduced per unit intervals
is constant. For example, we might want an error of 0.001 per unit time so
that from t = 0 to t = 10, the error is 0.01.
The adaptive step size method to do this has two parts:

1. Estimate error

2. Compare error to desired accuracy, and then increase/decrease step

size accordingly.

98
The idea is to choose some initial value of h, and then use ordinary Runge-
Kutta to do two steps of the algorithm, each of size h. Then after two time
steps, we estimate x(t + 2h). Then we go back to time step t and do Runge-
Kutta of size 2h. The estimate will generally be different from the previous
one.
How does this help? Well recall that RK is accurate to 4 th order with
5 order error. So the size of the error is ch5 for some c.
th

Starting at time t, and then doing two steps, the error will be roughly
2ch , i.e.,
5

x(t + 2h) = x1 +2ch5 .

|{z}
estimate

For a single large step, the error is of size 2h. Then c(2h)5 = 32ch5 , and
x(t + 2h) = x2 + 32ch5 . So the per-step error ε = ch5 is
1
ε = ch5 = (x1 − x2 ) .
30
We want ε to be some target accuracy. It might be better or worse, and we
can adapt it to make it close to our desired accuracy. So we ask ‘what is
the step size necessary to make the error size equal to the target accuracy?’
Let h0 denote the perfect step size. Taking steps of this size gives error
ε0 , where
 5  5
0 0 5 h0 1 h0
ε = c(h ) = ch 5
= (x1 − x2 ) .
h 30 h

Suppose the desired overall accuracy per unit time is δ. Then the desired
accuracy for a single step of size h0 is h0 δ. So solving for h0 , we’ll have
 0 5
1 h
h0 δ =
(x1 − x2 )
30 h
 1/4
30hδ
=⇒ h0 = h
|x1 − x2 |
≡ hρ1/4 ,

where we defined ρ to be the ratio of target accuracy to desired accuracy.

Then the complete method is as follows:

1. Perform two RK steps of size h. Perform one RK step of size 2h. Gives
estimates x1 and x2 for x(t + 2h).

99
 2. If ρ > 1, then the target accuracy is larger than the actual accuracy.
Make the step bigger for next time: set h0 = hρ1/4 .

3. If ρ < 1, then the actual accuracy is larger than the target accuracy.
Repeat calculation with a smaller step size: size h0 = hρ1/4 .
What are the pros and cons of this method? Well, the method means
more coding time, but will often result in decreased computational time. It
could also happen that x1 and x2 are very close, causing h0 to be unusually
large. The fix in this case would to be place an upper limit on how large
h0 can be. Also note that it’s import to repeat steps that miss the target
accuracy, otherwise errors built up.

Local extrapolation
In fact, we can improve the adaptive method slightly. Consider that


x(t + 2h) = x1 + 2ch5 + O h6 .

We estimated ch5 as (x1 − x2 )/30. Then add this, to get

1

x(t + 2h) = x1 + (x1 − x2 ) + O h6 .
15
This is accurate to order h5 , with error order h6 . It’s called local extrapola-
tion, and it comes for free. We can continue doing this, similar to what we
did with Romberg integration (and we’ll return to it later).

4.2.2 Other variations: leap-frog method

Consider
dx
= f (x, t).
dt
Recall in second-order RK, given x(t), we would estimate the value at t + h
using the slope at the midpoint f (x(t+h/2), t+h/2). And we would estimate
x(t + h/2) using Euler’s method. We can write these equations as
1
x(t + h/2) = x(t) + hf (x, t)
2    
1 1
x(t + h) = x(t) + hf x t + h , t + h .
2 2
The RK method looks like: estimate value at midpoint, and then use that
to get x(t + h).

100
The leap-frog method is the following variation:

where to get the next midpoint, we use the previous midpoint rather than
the endpoint. It starts out the same as RK, but then changes the way that
midpoints are calculated. In other words,
   
3 1
x t + h = x t + h + hf (x(t + h), t + h)
2 2
then get the next full step via
   
3 3
x(t + 2h) = x(t + h) + hf x t + h , t + h .
2 2
Basically, it amounts to repeatedly applying the equations
x(t + h) = x(t) + hf (x(t + h/2), t + h/2)
x(t + 3h/2) = x(t + h/2) + hf (x(t + h), t + h).
Each step
‘leaps over’ the previously

calculated value. Each step is accurate
to O h , and has error O h . The advantage of this method is that it’s
2 3

time-reversal symmetric.

4.3 Lecture 18: Leap frog & energy conservation,

Verlet method, modified midpoint method, Bulirsch-
Stoer technique
Recall the leap-frog method,

101
The equations were

x(t + h) = x(t) + hf (x(t + h/2), t + h/2)

x(t + 3h/2) = x(t + h/2) + hf (x(t + h), t + h) (?)

We claimed at the end of the last lecture that the method is time reversal
symmetric. Substitute h → −h and get:

x(t − h) = x(t) − hf (x(t + h/2), t − h/2)

x(t − 3h/2) = x(t − h/2) − hf (x(t − h), t − h).

Then shift t → t + 3h/2 and get

x(t + h/2) = x(t + 3h/2) − hf (x(t + h), t + h)

x(t) = x(t + h) − hf (x(t + h/2), t + h/2) (4)

Comparing 4 to ?, we can see the equations are the same, but run back-
wards! This is not true of RK, where you can do the same calculation and
you won’t have the time reversal symmetry.
Why is having this time reversal symmetry important? One reason is
conservation of energy. Recall that when using RK for the nonlinear pen-
dulum, the total energy fluctuated, and drifted over time. For the leap-frog
method, the total energy fluctuates (numerical error), but there is no drift!
So the leap-frog method is useful for solving energy conserving physical sys-
tems over long periods of time.
Let’s code it and see if this is true (use same preliminary lines as before
in the pendulum):

h = ( tMax − t 0 ) /1 e4
t = np . a r a n g e ( t0 , tMax , h / 2 ) #add h a l f −d i v i s i o n s
t h e t a = np . z e r o s ( np . s i z e ( t ) )
omega = np . copy ( t h e t a )
g , l = 9.81 , 0.5

f o r i i n r a n g e ( np . s i z e ( t ) ) :
i f i == 0 :
t h e t a [ i ] , omega [ i ] = t h e t a 0 , omega0
e l i f i == 1 : #E u l e r f i r s t h a l f −s t e p g u e s s
t h e t a [ i ] = t h e t a [ i −1] + ( 1 / 2 ) ∗h∗ fTheta ( omega [ i −1])
e l i f i == 2 : #4 th o r d e r RK f o r f i r s t f u l l −s t e p g u e s s
k1 = h∗ f T h e t a ( omega [ i −2])
k2 = h∗ f T h e t a ( omega [ i −2]+k1 / 2 )
k3 = h∗ f T h e t a ( omega [ i −2]+k2 / 2 )

102
 k4 = h∗ f T h e t a ( omega [ i −2]+k3 )
t h e t a [ i ] = t h e t a [ i −2] + ( 1 / 6 ) ∗ ( k1 + 2∗ k2 + 2∗ k3 + k4 )
K1 = h∗fOmega ( t h e t a [ i −2] , g , l )
K2 = h∗fOmega ( t h e t a [ i −2]+K1/ 2 , g , l )
K3 = h∗fOmega ( t h e t a [ i −2]+K2/ 2 , g , l )
K4 = h∗fOmega ( t h e t a [ i −2]+K3 , g , l )
omega [ i ] = omega [ i −2] + ( 1 / 6 ) ∗ (K1 + 2∗K2 + 2∗K3 + K4)
else : #l e a p f r o g
t h e t a [ i ] = t h e t a [ i −2] + h∗ fTheta ( omega [ i −1])
omega [ i ] = omega [ i −2] + h∗fOmega ( t h e t a [ i −1] , g , l )
plt . plot ( t , theta )

yields

which honestly is something I can be happy about. Before, when we saw

the total energy fluctuating upward, we could tell that our simulation was
basically not accurate enough to make physical sense. Now it is, and it does.

4.3.1 Verlet method

A variation of the leap frog method. Suppose the equations of motion take
the form
d2 x
= f (x, t).
dt2

103
For instance, F = ma. We can convert this to
dx dv
= v, = f (x, t).
dt dt
We could then directly apply the leap frog method to the vector r = (x, v),
with
dr
= f (r, t).
dt
But let’s instead write out the leap frog method in full:

x(t + h) = x(t) + hf (x(t + h/2), t + h/2), or

x(t + h) = x(t) + hv(t + h/2)
x(t + 3h/2) = x(t + h/2) + hf (x(t + h), t + h), or
v(t + 3h/2) = v(t + h/2) + hf (x(t + h), t + h).

We can derive a full solution using these two rewritten equations alone.
The vector method requires double the work, but here we only calculate x
at t + kh and v at t + ( 12 + k)h. This works for any diffeq with the special
form
dx dv
= v
|{z} , = f (x, t).
dt dt
does not depend
on x

One potential issue rises if we want to calculate a quantity that depends on

x and v, say for instance the total energy. We only have x at t + kh, and v
at t + (k + 12 h). Supposing that we did know v(t + h), then we could calculate
v(t + 12 h) by going backwards with the Euler method. This gives:
 
1 1
v t + h = v(t + h) − hf (x(t + h), t + h)
2 2

which we rearrange as
 
1 1
v(t + h) = v t + h + hf (x(t + h), t + h) .
2 2

So the full Verlet method (which is useful if we want to calculate quantities

104
dependent on both x, v at a given fixed time) is
 
1
x(t + h) = x(t) + hv t + h
2
k = hf (x(t + h), t + h)
 
1 1
v(t + h) = v t + h + k
2 2
   
3 1
v t + h = v t + h + k.
2 2
Simply bold the quantities x, v, k to get vector quantities.

4.3.2 Modified midpoint method

Another advantage of the leap frog method is that the total error is an even
function of the step size h, again due to the time reversal symmetry. This
means that expansion of the error in a power series of h has only even terms,
and no odd terms.
We can see this
in more detail: a single step
of the leap frog method
is accurate to O h , and has no error to O h . Write the error as ε(h),
2 3

with first term proportional to h3 . What do the other terms look like?
Take a small step forward with the leap frog method. This gives the
solution, plus ε(h), the error. Now go backwards, with step size −h. Due
to time symmetry, the change in the solution is the reverse of the forward
change. Thus the backward error is the negative of the forward error, i.e.

ε(−h) = −ε(h).

Well ε(h) is an odd function, and only has odd powers in the Taylor expan-
sion.
If we want the overall error, we can compute the error on a single step,
and then multiply by the number of steps. So if the time interval of interest
is Δ, then the number of steps is Δ/h, and the total error is ε(h) ∙ Δ/h. In
other words, this is an even function of the error, with first term proportional
to h2 .
However, there’s a slight catch: for the first step of the leap frog method,
we take a 1/2 step using the Euler method, and introduce an error of order h2
(as is the case with the Euler method). However, the Euler method’s higher
order terms are not even. So the total error has even and odd powers.
We can solve this problem using the modified midpoint method.
Suppose that we want to solve starting at t, and ending at t + H where H

105
is not small. Then use n steps of size h = H/n. We can write the leap frog
method as

x0 = x(t)
1
y1 = x0 + hf (x0 , t)
2
and then
 
1
x1 = x0 + hf y1 , t + h
2
y2 = y1 + hf (x1 , t + h)
 
3
x2 = x1 + hf y2 , t + h ,
2

etc. The general form for the midpoint method is

ym+1 = ym + hf (xm , t + mh) ,

 
1
xm+1 = xm + hf ym+1 , t + (m + )h
2

and the last two points are

 
1
yn = x t + H − h
2
xn = x(t + H).

Usually, we would take xn as the solution, we can also estimate x(t + H)

using the Euler method as
1
x(t + H) = yn + hf (xn , t + H).
2
We can take the average of the two estimates for x(t+H) to get the estimate:
 
1 1
x(t + H) = xn + yn + hf (xn , t + H) .
2 2

We can show that doing so cancels out the odd order error terms introduced
in the first step of Euler’s method. We need to track the errors carefully to
see this.

106
4.3.3 Bulirsch-Stoer Method
This method is reminiscent of Romberg integration (which I definitely slacked
on in the early notes). Suppose that we want to solve

dx
= f (x, t),
dt
from t to t + H. Begin by using a single step of size H, and then use the
modified midpoint method.
Let h1 = H. This gives an estimate of x(t + H), call it R1 . Then go back
to time t, and divide the interval into 2 steps of size h2 = 12 H. The gives
another estimate of x(t + H), call it R2 . Since the total error of the modified
midpoint method is an even function of step size, we have that


x(t + H) = R2,1 + c1 h22 + O h42

where c1 is some constant. Also consider that

x(t + H) = R1,1 + c1 h21 + O h41


= R1,1 + 4c1 h22 + O h42

using h1 = 2h2 . Then since both of these are equal to x(t + H), equate them
to get
1

c1 h22 = (R2,1 − R1,1 ) + O h42 .
3
Substitute into the above to get
1

x(t + H) = R2,1 + (R2,1 − R1,1 ) + O h42 .
3
This method then has an error of order h42 ; call the new estimate

1
R2,2 = R2,1 + (R2,1 − R1,1 ).
3
We can continue with this idea. Increase the number of step sizes to 3,
so that h3 = H/3. Solve from t to t + H to get the estimate R3,1 . Calculate
the estimate
4
R3,2 = R3,1 + (R3,1 − R2,1 ),
5

107
where the reasoning the same as before. We can then write


x(t + H) = R3,2 + c2 h43 + O h63 ,
where c2 is some constant. From before we have that


x(t + H) = R2,2 + c2 h42 + O h62
81

= R2,2 + c2 h43 + O h62 ,
16
using h2 = 3h3 /2. Equating and rearranging then gives
16
c2 h43 = (R3,2 − R2,2 ).
65
Substituting back in gives


x(t + H) = R3,3 + O h63 ,
where R3,3 = R3,2 + 1665 (R3,2 − R2,2 ). This result is accurate to order h3 , with
6

only three steps!

The power here is in the cancellation of higher order errors on successive
steps. This is a result of doing the procedure with the modified midpoint
method, which has only even-order terms. We can continue this process
iteratively, and the estimate is:


x(t + H) = Rn,m+1 + O hn2m+2 ,
where m < n and
Rn,m − Rn−1,m
Rn,m+1 = Rn,m + .
[n/(n − 1)]2m − 1
The diagram winds up being similar to that for Romberg integration:

108
Moreover, we also get an estimate of the error as
Rn.m − Rn−1,m
.
[n/(n − 1)]2m − 1

Some limitation are that

1. The answers are only accurate for x(t + H).

2. The method only converges quickly if the power series expansion of

x(t + H) converges quickly.

We can overcome these problems by dividing the interval into smaller ones,
and applying the Bulirsch-Stoer method to each one.

4.4 Lecture 19: Boundary value problems, shoot-

ing method, partial diffeqs, finite differences
Let’s start by talking about boundary value problems. For instance, the
differential equation for the height of a ball thrown in the air is

d2 x
= −g.
dt2
We could specify the initial condition in order to fix a solution, for instance,
the initial height and velocity. Alternatively, we could specify that the ball
has initial height x = 0 at t = 0 and final height x = 0 at t = t1 , where
t1 > 0. The goal is to find a solution that satisfies these conditions.

4.4.1 Shooting method

One way to achieve the above goal. Idea: just try a number of possibilities
until we find a solution. For the above example, just pick an initial velocity,
solve the differential equation numerically, and then see if the final time and
height are as desired.
The key question then is: how do we modify our guesses to ensure con-
vergence? Consider that there is some function f such that

x(t1 ) = f (v0 ).

The height at time t1 is a function of the initial velocity, and we would like
to find v0 such that f (v0 ) satisfies the above condition.

109
 So we can just use a root finding method, such as binary search. For the
example, consider that we can rewrite this as
dx dy
= y, = −g.
dt dt
We can then solve using 4th order RK. Then perform a binary search for
the initial velocity. This is visible in the online code 1-throw.py, and I feel
comfortable enough with what to do to not write it myself.

When every term is linear in the dependent variable...

A special kind of BVP occurs for diffeqs such that every term is linear in
the dependent variable. For example, consider the Schrödinger equation

~ 2 d2 ψ
− + V (x)ψ(x) = Eψ(x),
2m dx2
where ψ is the wavefunction, V is the potential energy, and E is the total
energy of the particle. Suppose we have a square well potential
(
0 for 0 < x < L
V (x) =
∞ else.

We can solve this problem analytically, but let’s explore some numerics. The
probability of finding a particle at x where V (x) = ∞ is zero, so ψ(x) = 0
at x = 0, x = L. Since the equation is second order, rewrite it as two first
order ones:
dψ dφ 2m
= φ, = 2 [V (x) − E] ψ.
dx dx ~
Then we need initial conditions, ψ(x) = 0 at x = 0, but we do not know

dψ 
φ(0) = .
dx x=0

Well, let’s just guess a value! Then solve, and check if the final ψ(x = L) = 0
boundary condition is satisfied. Upon first guessing, it’s quite likely then
that the condition will not be satisfied.
We could try using the shooting method, but in this case it won’t work

(why?). Suppose that we change the initial value dψ dx x=0 by doubling it.
Since Shrödinger’s equation is linear, this doubling leads to a double of the
solution ψ(x) which just means that the endpoint ψ(L) will get scaled by 2.
For instance, suppose the initial solution was

110
then doubling the slope would lead to

The problem here would be that there is no solution to the equation for
this energy E. There are only solutions for particular values of the energy
parameter E, which is why energy is quantized.
So change the goal to be: find the allowed energies. Start with some
initial conditions, and the vary E to find the value of which ψ(0) = 0, and
ψ(L) = 0. We can think that Schrödinger’s equation gives a function f (E)

111
equal to the wavefunction at x = L, and the goal is to find E such that
f (E) = 0 (and then we can use a root-finding method). 

But then what about the boundary condition for dψ dx x=0 ? It doesn’t
matter, since it will just scale the solution. Just pick it so that
Z
|ψ(x)|2 dx = 1,

i.e. so that the wavefunction is normalized.

This is all done in 2-squarewell.py, which again we don’t have access
to. However (and I definitely realized this too late) Mark Newman provides
some sample programs for the later chapters of his book here, so looking at
those is also an option.

4.4.2 Chapter 9: Partial differential equations.

There are many examples of PDEs in physics: the wave equation, the diffu-
sion equation, the Laplace equation, Poisson equation, Maxwell’s, Schrödinger’s,
and more. Let’s begin with boundary value problems.

Example: consider a box with conducting walls. The goal is to find the
electrostatic potential inside the box. Say three walls are grounded, and the
top is at potential V .

The potential φ is related to the vector E-field by E = −∇φ. In the absence

of charges, the Maxwell equation states that ∇ ∙ E = 0. This them yields

∇2 φ = 0 (Laplace’s equation).

112
or
∂2φ ∂2φ ∂2φ
+ 2 + 2 = 0.
∂x2 ∂y ∂z
We want to solve this subject to the top being V , and the others being 0.

4.4.3 Method of finite differences

We’ll restrict the problem to 2D so that we can easily plot the solution. So
we have
∂2φ ∂2φ
+ 2 = 0.
∂x2 ∂y
Divide up the space into a regular square grid. Put points on the boundary
(where we know the solution) and inside (where we don’t). Let a be the
spacing of the grid. Then recall the central difference approximation for
second partial derivatives:

∂2φ φ(x + a, y) + φ(x − a, y) − 2φ(x, y)

2
≈ .
∂x a2
It’s an approximation in terms of three neighboring points. There’s a similar
2
equation for ∂∂yφ2

∂2φ φ(x, y + a) + φ(x, y − a) − 2φ(x, y)

2
≈
∂y a2
then add these to get
 
∂2φ ∂2φ 1 φ(x + a, y) + φ(x − a, y) + φ(x, y + a)
+ 2 ≈ 2 .
∂x 2 ∂y a +φ(x, y − a) − 4φ(x, y)

We can represent this visually as

113
Then, rearranging gives us

0 = φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a) − 4φ(x, y).

This is now a set of simultaneous linear equations which we can solve in

principle, using Gaussian elimination (for instance). But we could also use
the relaxation method. In this case, rewrite the problem as
1
φ(x, y) = [φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a)]
4
so that φ(x, y) is just a weighted average of adjacent points. This particular
method is called the Jacobi method. In fact, it’s possible to prove that in
this scenario, it converges. We can set it to stop iterating after some target
accuracy is reached.
Bring up laplace.py to see this (n.b., this target accuracy took a couple
minutes for my laptop):
from numpy import empty , z e r o s , max
from pylab import imshow , gray , show

# Constants
M = 100 # Grid s q u a r e s on a s i d e
V = 1.0 # V o l t a g e a t top w a l l
t a r g e t = 1 e−6 # Target a c c u r a c y

# Create arrays to hold p o t e n t i a l values

p h i = z e r o s ( [M+1 ,M+1] , f l o a t )
phi [ 0 , : ] = V
p h i p r i m e = empty ( [M+1 ,M+1] , f l o a t )

# Main l o o p
delta = 1.0
w h i l e d e l t a >t a r g e t :

# C a l c u l a t e new v a l u e s o f t h e p o t e n t i a l
f o r i i n r a n g e (M+1) :
f o r j i n r a n g e (M+1) :
i f i==0 o r i==M o r j==0 o r j==M:
phiprime [ i , j ] = phi [ i , j ]
else :
p h i p r i m e [ i , j ] = ( p h i [ i +1, j ] + p h i [ i −1, j ] \
+ p h i [ i , j +1] + p h i [ i , j −1]) /4

# C a l c u l a t e maximum d i f f e r e n c e from o l d v a l u e s
d e l t a = max( abs ( phi −p h i p r i m e ) )

# Swap t h e two a r r a y s around

114
 phi , p h i p r i m e = phiprime , p h i

# Make a p l o t
imshow ( p h i )
gray ( )
show ( )

Figure 4.2: Solution φ(x, y) of Laplace equation on grid with φ = 0 at the

left, right, and bottom boundaries, and φ = V fixed at the upper boundary.

A couple things to note: first, realize that the Jacobi method is just an
approximation. We can increase its accuracy by increasing the number of
grid points, or by using higher order derivative approximations.
Also, we need to be more clever in cases where the boundary conditions
are not as simple. In such cases, we might need to pick nonuniform spacing
for grid points.

115
Example: Poisson equation, electrostatics. Recall the Poisson equa-
tion:
ρ
∇2 φ = − ,
ε0
for ρ the charge density in space. In this case, make all the walls have zero
potential, but put some charges inside the box:

And let the charge density be 1Cm −2 . Again, we can make use of the
relaxation method. Write the Poisson equation as:
1 ρ(x, y)
2
[φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a) − 4φ(x, y) ] = − .
a ε0
Then rewrite it as
1 a2
φ(x, y) = [φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a)] + ρ(x, y).
4 4ε0
Bring up 4-poisson.py to see this in action (or, if you’re me, butcher the
previous Laplace code to see it):
from pylab import imshow , gray , show

# Constants
M = 100 # Grid s q u a r e s on a s i d e
t a r g e t = 1 e−5 # Target a c c u r a c y
rho0 = 1 # Charge d e n s i t y (C/m∗ ∗ 2 )
l e n = 14 # Length o f s q u a r e c h a r g e (cm)
pX , pY= 7 0 , 50 # P o s i t i v e charge c e n t e r coord
nX , nY = 3 0 , 50 # Negative charge c e n t e r coord

# Create arrays to hold p o t e n t i a l values

p h i = z e r o s ( [M+1 ,M+1] , f l o a t )

116
p h i p r i m e = empty ( [M+1 ,M+1] , f l o a t )
# Create array to hold values o f charge d e n s i t y
rho = z e r o s ( [M+1 ,M+1] , f l o a t )
rho [ pX−l e n / 2 :pX+l e n / 2 , pY−l e n / 2 :pY+l e n / 2 ] = rho0
rho [ nX−l e n / 2 :nX+l e n / 2 , nY−l e n / 2 :nY+l e n / 2 ] = −rho0

# Main l o o p
delta = 1.0
w h i l e d e l t a >t a r g e t :

# C a l c u l a t e new v a l u e s o f t h e p o t e n t i a l
f o r i i n r a n g e (M+1) :
f o r j i n r a n g e (M+1) :
i f i==0 o r i==M o r j==0 o r j==M:
phiprime [ i , j ] = phi [ i , j ]
else :
p h i p r i m e [ i , j ] = ( p h i [ i +1, j ] + p h i [ i −1, j ] \
+ p h i [ i , j +1] + p h i [ i , j −1]) /4 \
+ rho [ i , j ] / ( 4 )

# C a l c u l a t e maximum d i f f e r e n c e from o l d v a l u e s
d e l t a = max( abs ( phi −p h i p r i m e ) )

# Swap t h e two a r r a y s around

phi , p h i p r i m e = phiprime , p h i
print ( delta )

# Make a p l o t
imshow ( p h i )
gray ( )
show ( )

where we decided to use slightly different coordinates for a nicer-looking

plot:
Obviously, from using it a few times we can see that convergence for
the Jacobi method is nail-bitingly slow. How do we speed it up? With the
overrelaxation method. The idea here is to overshoot the target value
by a little. Suppose that we iterate, beginning with φ(x, y), so that the next
iteration gives φ0 (x, y). Then φ0 (x, y) = φ(x, y) + Δφ(x, y), where Δ is the
change. Then for overrelaxation, we do

φw (x, y) = φ(x, y) + (1 + w)Δφ(x, y),

where ω > 0. Substituting this in then gives:

 
φw (x, y) = φ(x, y) + (1 + w) φ0 (x, y) − φ(x, y)
= (1 + w)φ0 (x, y) − wφ(x, y).

117
 Figure 4.3: Solving the Poisson equation ∇2 φ = ρ,
for φ(x, y) with two opposite charge densities.

Then for the example of Laplace’s equation, we get

1+w
φw (x, y) = [φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a) ] − wφ(x, y).
4
How do we choose w? Answer: pick w < 1. There will be more discussion
in Lecture 20...

118
4.5 Lecture 20: Gauss-Seidel, initial value prob-
lems, forward time centered space method,
numerical stability analysis.
Last lecture to transcribe (although for what it’s worth, I think we’re just
getting started with computational physics in Python. As soon as we wrap
this up, we’re going to shift focus to bonafide problem-solving. We’ve seen
a bunch of techniques, now it’s time to use them. After we’ve solved a
couple of the homework problems, we can go ahead and check out this GUI
tutorial).
Now, from last time, we introduced overrelaxation as a variation of the
finite difference or Jacobi method. Recall that an iteration is

φ0 (x, y) = φ(x, y) + Δφ(x, y).

The idea for overrelaxation was

φw (x, y) = φ(x, y) + (1 + w)Δφ(x, y),

with w > 0. For the Laplace equation solution from before, this translates
to
(1 + w)
φw (x, y) = [φ(x + a, y) + φ(x − a, y) + φ(x, y + a) + φ(x, y − a)] − wφ(x, y).
4
Consider then, a slightly different method:

4.5.1 Gauss-Seidel method (GS)

In the previous method, we need an old array and a new array, and then
use all the terms of the old array to make updates. When doing this, we
have to scan elements one-by-one, and process them. The idea for GS is to
simply use one array, and then use the newest values for updating. So the
method looks like this:
for x in grid
for y in grid
p h i ( x , y ) =(1/4) ∗ [ p h i ( x+a , y ) +
p h i ( x−a , y ) +
p h i ( x , y+a ) +
p h i ( x , y−a ) ]]

119
We can then combine GS with the overrelaxation method to have the update
be:
1+w
φ(x, y) ← [φ(x + a, y) + ... ] − wφ(x, y).
4
The advantage of GS is that it uses less memory. Moreover, GS with over-
relaxation is stable, but overrelaxation alone is not. The method is proven
stable, provided w < 1.

4.5.2 Initial value problems

A good proportion of physics problems (especially dynamical ones) are ap-
proximately this: given some starting conditions, what happens in the fu-
ture?

Example: diffusion equation. The diffusion equation is

∂φ ∂2φ
= D 2,
∂t ∂x
where D is the diffusion constant. This equation is used to calculate the
motion of diffusing gases and liquids, as well as the flow of heat in thermal
conductors. It’s a PDE, and as we’ve expressed it there are two indepedent
variables.
Divide the spatial dimension into a line of points along the x axis (evenly
spaced points; spacing a). Then write

∂2φ φ(x + a, t) + φ(x − a, t) − 2φ(x, t)

2
≈ ,
∂x a2
which means that we’re trying to solve
∂φ D
= 2 [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)] .
∂t a
This gives a set of simultaneous ordinary differential equations, but in φ(x, t), φ(x+
a, t), φ(x − a, t). We can solve these using the methods from before.
But then ask: which method should we use? It actually makes most
sense to use the Euler method here. This is because the approximation to
the second-derivative given above has a second-order error. So it doesn’t
make any sense to use more computationally intensive RK, which has an
input with a higher error. The error in the Euler method is second-order,

120
which is comparable to the error introduced by approximating the second
derivative with the central-difference formula above.
Recall then that the Euler method is for solving
dφ
= f (φ, t).
dt
Then Taylor expand φ(t) about time t:

dφ 
φ(t + h) ≈ φ(t) + h + ...
dt t
= φ(t) + hf (φ, t).

Applying this to our case gives

D
φ(x, t + h) = φ(x, t) + h [φ(x + a, t) + φ(x − a, t) − 2φ(x, t) ] ,
a2
where knowing every grid value at a given time allows us to predict all the
next times. This method is called forward time centered space (FTCS).
It also winds up being very strong memory-wise to code, since you only need
to have two arrays (φ and φ0 in this case) store the data as you propagate
in time. You can then print/plot specific times to see the evolution of your
system without crashing your computer.

Example: temperature profile of a container made of stainless

steel, and submerged in two different baths.

Also, let the container be 1cm deep in the outer cold water. The thermal
conduction here is governed by the heat equation

∂T ∂2T
=D 2.
∂t ∂x
So then we make a grid, and solve it, assigning D = 4.25×10−6 m2 s−1 . Bring
up 1-heat.py to see this in action:

121
from numpy import empty
from pylab import p l o t , x l a b e l , y l a b e l , show

# Constants
L = 0.01 # Thickness o f s t e e l in meters
D = 4 . 2 5 e−6 # Thermal d i f f u s i v i t y
N = 100 # Number o f d i v i s i o n s i n g r i d
a = L/N # Grid s p a c i n g
h = 1 e−4 # Time−s t e p
e p s i l o n = h /1000

Tlo = 0 . 0 # Low t e m p e r a t u r e i n C e l c i u s
Tmid = 2 0 . 0 # Intermediate temperature in Celcius
Thi = 5 0 . 0 # Hi t e m p e r a t u r e i n C e l c i u s

t1 = 0.01
t2 = 0.1
t3 = 0.4
t4 = 1.0
t5 = 10.0
tend = t5 + e p s i l o n

# Create arrays
T = empty (N+1 , f l o a t )
T [ 0 ] = Thi
T [N] = Tlo
T [ 1 : N] = Tmid
Tp = empty (N+1 , f l o a t )
Tp [ 0 ] = Thi
Tp [N] = Tlo

# Main l o o p
t = 0.0
c = h∗D/ ( a ∗ a )
w h i l e t<tend :

# C a l c u l a t e t h e new v a l u e s o f T
f o r i i n r a n g e ( 1 ,N) :
Tp [ i ] = T [ i ] + c ∗ (T [ i +1]+T [ i −1]−2∗T [ i ] )
T, Tp = Tp , T
t += h

# Make p l o t s a t t h e g i v e n t i m e s
i f abs ( t−t 1 )<e p s i l o n :
p l o t (T)
i f abs ( t−t 2 )<e p s i l o n :
p l o t (T)
i f abs ( t−t 3 )<e p s i l o n :
p l o t (T)

122
 i f abs ( t−t 4 )<e p s i l o n :
p l o t (T)
i f abs ( t−t 5 )<e p s i l o n :
p l o t (T)

Note that starting out, the profile is biased, but then it becomes more and
more linear as time goes on. Also, we could also implement time-varying
boundary conditions.

4.5.3 Numerical stability analysis

Recall how we mentioned that FTCS fails for the wave equation. Let’s
analyze why...
Recall that the wave equation is

∂2φ 1 ∂2φ
= .
∂x2 v 2 ∂t2
To solve it using the FTCS method, we deivide space into discrete points
with spacing a:

∂2φ v2
= [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)] .
∂t2 a2

123
When then divide this into two simultaneous first order equations:
∂φ
= ψ(x, t),
∂t
∂ψ v2
= 2 [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)]
∂t a
Right. So then applying Euler’s method gives

φ(x, t + h) = φ(x, t) + hψ(x, t)

v2
ψ(x, t + h) = ψ(x, t) + h [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)]
a2
and the idea is then to iterate these equations. What will happen? Well, over
different time scales, we see different things (see Figure below). Basically,
over long time scales, the calculation becomes numerically unstable. Why?

124
 We can get an idea by applying a von Neumann stability analysis. Return
to the diffusion equation for a moment (in FTCS form):

hD
φ(x, t + h) = φ(x, t) + [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)] .
a2
Assuming that the spatial region is finite, we can express φ(x, t) with a
Fourier expansion:
X
φ(x, t) = ck (t)eikx .
k

where we’re using wave-vectors k and coefficients ck (t). This is a useful

approach because the diffusion equation is linear. If we plug in a single term
ck (t)eikx , we get

hD h i
φ(x, t + h) = ck (t)eikx + 2 ck (t) eik(x+a) + eik(x−a)
 a i
hD h ika −ika
= ck (t)e ikx
1+ 2 e +e −2
a
  
4D ka
= ck (t)eikx 1 − h 2 sin2 .
a 2

Note that for the final expression, we used the identities eiθ + e−iθ = 2 cos θ,
and 1 − cos θ = 2 sin2 (θ/2).
What this means is that each term in the Fourier series transforms in-
dependently. The terms don’t couple. Thus
  
4D 2 ka
ck (t + h) = 1 − h 2 sin ck (t).
a 2

So now we see that for stability, we require that the prefactor has magnitude
≤ 1. So we need
  
1 − h 4D sin2 ka  ≤ 1.
 
 a 2 2 

Well, we always have

 
4D ka
1 − h 2 sin2 ≤ 1.
a 2
| {z }
because is
always ≥0

125
So then we need
 
4D 2 ka
−1 ≤ 1 − h 2 sin
a 2
 
4D ka
=⇒ 2 ≥ h 2 sin2 .
a 2

So it’s sufficient to take h4D/a2 ≤ 2, or

2a2
h≤ .
4D
If strict inequality holds, then all terms in the Fourier series, except for k = 0,
converge exponentially fast to zero. This implies that the final solution is
uniform in space, which is what we expect for the diffusion equation.

Returning to the wave equation, we see that it’s slightly more com-
plicated because we have coupled equations. However, we can expand the
vector

   
φ(x, t) cφ (t) ikx
→ e .
ψ(x, t) cψ (t)

Plugging into the evolution equations from before, we get

cφ (t + h) = cφ (t) + hcψ (t)

 
4v 2 ka
cψ (t + h) = cψ (t) − hcφ (t) 2 sin2
a 2

Writing it as a matrix-vector equation,

ˉ
c(t + h) = Ac(t)

where
 
cφ (t)
c(t) = ,
cψ (t)
 
1 h
Aˉ = ,
−hr2 1
2v ka
r= sin .
a 2

126
Nice. At each time step, c gets multiplied by a 2×2 matrix that depends only
on k. Then we can write c(t) as a linear combination of the two eigenvectors
ˉ
of A:

c(t) = α(t)v 1 + β(t)v 2 ,

ˉ
=⇒ c(t + h) = A(α(t)v 1 + β(t)v 2 )
= α(t)λ1 v 1 + β(t)λ2 v 2 .

Repeating this again gives

c(t + 2h) = α(t)λ21 v 1 + β(t)λ22 v 2 ,

and for m steps, gives

1 v 1 + β(t)λ2 v 2 .
c(t + mh) = α(t)λm m

Well, the eigenvalues for Aˉ are λ = 1 ± ihr. So the magnitude for both is
s  
p 4h2 v 2 2 ka
|λ| = 1 + h r = 1 +
2 2 sin .
a2 2

In other words, it’s never less than one! By our previous analysis, this means
the FTCS method is never stable for the wave equation!
How can we solve this problem? We do it using the ‘implicit’ method.
Basically, substitute hw − h in the evolution equations. This leads to

φ(x, t − h) = φ(x, t) − hψ(x, t)

v2
ψ(x, t − h) = ψ(x, t) + − [φ(x + a, t) + φ(x − a, t) − 2φ(x, t)] .
a2
These tell us how to go back in time. So substitute t → t + h, and then
rearrange to get

φ(x, t + h) − hψ(x, t + h) = φ(x, t)

v2
ψ(x, t + h) − h [φ(x + a, t + h) + φ(x − a, t + h) − 2φ(x, t + h) ] = ψ(x, t).
a2
This isn’t an explicit expression, but we can solve for future values with it.
It’ll also be slower. However, it is numerically stable.

127
 So then we can rewrite the equations as
ˉ
Bc(t + h) = c(t), where
 
ˉ 1 −h
B= .
hr2 1

This means that

ˉ −1 c(t)
c(t + h) = B
 
1 1 h
= c(t).
1 + h2 r2 −hr2 1

Great! Now the eigenvalues are

1 ± ihr 1
λ = , =⇒ |λ| = √ .
1 + h2 r 2 1 + h2 r 2
We have stability. However, we also know that what we have is unphysi-
cal! We want the waves to propagate indefinitely, but what we have is an
exponential decay.
This is where Mark Wilde’s notes leave off. Cliffhanger, much?

128
Chapter 5

Assessment and next steps

Okay. So we’ve come a good ways here. Our original goals were:

1. Learn how to approach and solve physics problems with computers,

2. Develop fluency with the BaKoMA TeX editor, and

3. Develop fluency with Python.

How have we done? We read and transcribed 20 lectures, and worked

through the majority of 4/5 homeworks. I’m basically just too lazy to do
the last one, and I want to get on to reassessing what I should be doing with
my time before the semester starts.
Vis-a-vis goal 2, I think we’re pretty well set. We can obviously write
coherent notes with BaKoMaTeX. Code integration, figure integration, as
well as custom-macros are all pretty well done. Writing math is pretty good,
but it’ll never be as abstract as free-hand on paper. This makes sense. If you
want to be thinking precisely, or summarizing something you (or someone
else) has thought, typesetting on BaKoMa will be a good tool. However,
if you want depth, and abstraction, and internalization, the risk here is
that the note-taking gains an upper-edge on the learning. In fact, it almost
certainly will – too much effort and focus will be devoted to ‘making the
notes pretty’, rather than focusing on the concepts and ideas.
So what? So – depending on the class / material, you’ll need a dif-
ferent default. For instance, the dual-program OneNote/BaKoMa, with a
focus away from transcription and towards being ‘idea-centered’ depending
on the material. The rule of thumb there will be: if it’s extremely idea-
dense, (in addition to being hard to typeset), then handwritten. And since
a good number of your classes will be idea dense, they’ll be hand-written.

129
Post-lecture, you can summarize or process those notes in this, probably at
something like 30mins-1hr/lecture (which is also good lecture review).
What about goals 1 and 3? Well let’s talk about the Python side. I
gained some real familiarity with Numpy and Matplotlib, not really Scipy
at all. The libraries read very similar to previous libraries I’ve worked with
(especially in JDFTx) which is nice. There are still obviously major parts
of just Numpy that I don’t have a good grasp on, and that are important for
being able to do basic tasks in Python (e.g., array manipulation, working
with dictionaries, creating UI’s).
But I like Python. I think that I’ll be able to keep working on it, and
at the same time, I’ll be able to learn physics through it. How do I mean?
Well, I see a new concept, or some new differential equation, and I’ll be able
to simulate it. That’s the point! Obviously, not a lot of textbooks or classes
are catered to numerics. That’s both the challenge and the opportunity –
it’s building myself into a so-called Π-shaped practitioner (instead of the
traditional ‘T’): I’ll need to develop real depth in both the field-specifics, as
well as in the numerical/computational realizations of those specifics.
I think it’s really telling that I could have gotten through my undergrad
without having ever seen something like the trapezoidal rule. Maybe it was
implicit in calculus, and so I saw it and ‘internalized’ it a long time ago.
But why did I never build on it? Why did no one ever mention you could
actually DO IT on a computer? And that you should WANT TO? Well,
this is about fixing that.
Finally, on ‘approaching and solving physics problems with computers’:
that’s obviously a high bar. But I think the methods I’ve seen will con-
tinue to be useful: time-propagating differential equations (e.g., with Runge-
Kutta) is immensely useful, as is being able to numerically evaluate integrals.
Spending some time on Fourier transforms was also important. Also, I ba-
sically wrote a computational solver to the time-independent Schrödinger
equation, for arbitrary potentials. That’s something you could never get
close to, analytically. Being able to manipulate arrays of data (e.g., Fourier
transforms, iterating operations, plotting) is also immensely practical.
Basically, there needs to be a balance. I think there’s a Python rabbit
hole that I’m only just beginning to jump down, but all the analytical meth-
ods I’ve seen aren’t useless. In fact, they’re essential to be able to do any
kind of independent scientific computation. And that kind of computation
is the right kind of challenge: it says ‘you should value being able to explain
things to computers, because they’ll help you understand more material, at
greater depth, and with greater efficiency’.

130