Topic 6: Inner Product Spaces

[AR Chap 6]

6.1 Definition of inner products

6.2 Geometry from inner products
6.3 Cauchy-Schwarz inequality
6.4 Orthogonality and projections
6.5 Gram-Schmidt orthogonalization procedure
6.6 Application: curve fitting

251

6.1 Definition of inner products

The Euclidean length of a vector in v Rn is defined as
q

kvk = v12 + v22 + + vn2 = v v

Similarly, the distance between vectors u and v is given by ku vk.

The angle between two vectors u and v was defined by
cos =

vu
kvkkuk

with

066

The projection of v onto u was given in terms of the dot product by

proju v =

(u v)
u
kuk2

252

We want to address two issues:

1. How to generalise the notion of a dot product for vector spaces
other than Rn .
2. To use geometry associated with the dot product and its
generalisations to study vector spaces and linear transformations.
The first can be done by looking carefully at some of the key properties
of the dot product and generalising them.

253

Definition (Inner Product)

Let V be a vector space over the real numbers. An inner product on V
is a function that associates with every pair of vectors u, v V a real
number, denoted hu, vi, satisfying the following properties.
For all u, v, w V and R:
1. hu, vi = hv, ui

2. hu, vi = hu, vi

3. hu + v, wi = hu, wi + hv, wi
4. a. hu, ui > 0
b. hu, ui = 0 u = 0

A vector space V together with an inner product is called an inner

product space.
p
Then we can define the length of u as kuk = hu, ui.
Note
A vector space can admit many different inner products. Well see some
examples shortly.
254

In words these axioms say that we require the inner product to:
1. be symmetric;
2. be linear with respect to scalar multiplication;
3. be linear with respect to addition;
4.

a. have positive squared lengths;

b. be such that only the zero vector has length 0.

With V = Rn , we of course have that hu, vi = u v defines an inner

product (the Euclidean inner product).
But this is not the only inner product in the vector space Rn .

255

Example
Show that, in R2 , if u = (u1 , u2 ) and v = (v1 , v2 ) then
hu, vi = u1 v1 + 2u2 v2
defines an inner product.

256

More examples
1. Show that in R3 , hu, vi = u1 v1 u2 v2 + u3 v3 does not define an
inner product by showing that axiom 4a does not hold.

2. Show that in R2 ,
 T
hu, vi = u



 
 2 1 v1
2 1   
v = u1 u2
1 1
1 1
v2

defines an inner product.

257

258

The last example raises

the

 question of what we need to know about a
 T  
a b
2 2 matrix A =
to ensure that hu, vi = u A v defines an
c d
inner product.
The check that this satisfies axioms 2 and 3 follows exactly the same
lines as the previous example.
In order to satisfy axiom 1 we will need to make b = c.
The hard work occurs in satisfying axiom(s) 4. A quick check with the
standard basis vectors shows that we must have a > 0 and d > 0.
A harder calculation, which involves completing a square, shows that
axiom(s) 4 will be satisfied exactly when det(A) > 0 (and a > 0).
So we can tell exactly when a 2 2 matrix A defines an inner product
in this way.
259

Another example
In the case of the vector space Pn of polynomials of degree less than or
equal to n, a possible choice of inner product is
Z 1
p(x)q(x) dx.
hp, qi =
0

To verify this, we would need to check, for any polynomials

p(x), q(x), r (x) and any scalar that
R1
R1
1. 0 p(x)q(x) dx = 0 q(x)p(x) dx
R1
R1
2. 0 p(x)q(x) dx = 0 (p(x))q(x) dx
R1
R1
R1
3. 0 (p(x) + q(x)) r (x) dx = 0 p(x)r (x) dx + 0 q(x)r (x) dx
R1
4. 0 p(x)2 dx > 0
R1
5. 0 p(x)2 dx = 0 only when p(x) is the zero polynomial

260

6.2 Geometry from inner products

In a general vector space, how can we measure angles and find lengths?
If we fix an inner product on the vector space, we can then define
length and angle using the same equations that we saw above for Rn .
We simply replace the dot product by the chosen inner product.

Definition (Length, Distance, Angle)

For a real vector space with an inner product h , i we define:
p
the length (or norm) of a vector v by kvk =
hv, vi

the distance between two vectors v and u by d(v, u) = kv uk

and the angle between v and u using

cos =

hv, ui
kvkkuk

with

066
261

Note
In order for this definition of angle to make sense we need
1 6

hv, ui
61
kvkkuk

We will see shortly that this is always the case

(the Cauchy-Schwarz inequality)

Definition (Orthogonal vectors)

Two vectors v and u are said to be orthogonal if hv, ui = 0

262

Example
h(u1 , u2 ), (v1 , v2 )i = u1 v1 + 2u2 v2 defines an inner product on R2
If u = (3, 1) and v = (2, 3), then
p

d(u, v) = kuvk = k(5, 2)k = h(5, 2), (5, 2)i = 25 + 8 = 33

and

hu, vi = h(3, 1), (2, 3)i = 3 (2) + 2 1 3 = 0

so u and v are orthogonal (using this inner product)

263

Example (an inner product for functions)

The set of all continuous functions
C [a, b] = {f : [a, b] R | f is continuous}
is a vector space.
Its a subspace of F([a, b], R), the vector space of all functions.
For f , g C [a, b] define
hf , g i =

f (x)g (x)dx
a

This defines an inner product.

264

Example
Consider C [0, 2] with the inner product hf , g i =

R 2
0

f (x)g (x)dx

The norms of the functions s(x) = sin(x) and c(x) = cos(x) are:
2

ksk = hs, si =

sin (x)dx =

2
0

1
(1 cos(2x))dx
2

2
1
x
=
sin(2x)
=
2 4
0

So ksk = and (similarly) kck =

hs, ci =

sin(x) cos(x)dx =

2
0


2
1
1
=0
sin(2x)dx = cos(2x)
2
4
0

So sin(x) and cos(x) are orthogonal

This is used in the study of periodic functions (using Fourier series) in
(for example) signal analysis, speech recognition, music recording etc.
265

6.3 Cauchy-Schwarz inequality

Theorem (Cauchy-Schwarz inequality)
Let V be a real inner product space.
Then for all u, v V
|hu, vi| 6 kukkvk
Further, equality holds if and only if one vector is a multiple of the
other.
Proof: The proof is as for the case V = Rn on slide 82.
Note: This implies that the definition of angle is okay!
We defined the angle between two vectors using
hu, vi
cos =
kukkvk
From the Cauchy-Schwartz inequality we know that
hu, vi
1 6
61
kukkvk

266

Application
For two continuous functions f , g : [a, b] R, it follows directly from
the Cauchy-Schwarz inequality that
Z

f (x)g (x)dx

2

Z

f (x)dx
a

 Z

g (x)dx
a

And we dont need to do any (more) calculus to prove it!

Set f (x) =
We obtain

x and g (x) = 1/ x; also a = 1, b = t > 1.

Z

1 dx

2

which becomes
 2

t 1
2
(t 1) 6
loge t
2

Z

x dx

or

 Z

1
dx
x

loge t >

2(t 1)
, t > 1.
t+1
267

6.4 Orthogonality and projections

[AR 6.2]

Orthogonal sets
Recall that u and v are orthogonal if hu, vi = 0

Definition (Orthogonal set of vectors)

A set of vectors {v1 , . . . , vk } is called orthogonal if hvi , vj i = 0
whenever i 6= j.

268

Examples
1. {(1, 0, 0), (0, 1, 0), (0, 0, 1)} is orthogonal in R3 with the dot
product as inner product.

2. So is {(1, 1, 1), (1, 1, 0), (1, 1, 2)} using the dot product.

3. {sin(x), cos(x)} is orthogonal in C [0, 2] equipped with the inner

product defined (using a definite integral) on slide 265.

269

Proposition
Every orthogonal set of nonzero vectors is linearly independent
Proof:

270

Orthonormal sets

Definition
A set of vectors {v1 , . . . , vk } is called orthonormal if it is orthogonal
and each vector has length one. That is
(
0 i 6= j
{v1 , . . . , vk } is orthonormal
hvi , vj i =
1 i =j
Note
Any orthogonal set of non-zero vectors can be made orthonormal by
dividing each vector by its length.
Examples
1. In R3 with the dot product:

{(1, 0, 0), (0, 1, 0), (0, 0, 1)} is orthonormal

{(1, 1, 1), (1, 1, 0), (1, 1, 2)} is not (though it is orthogonal)

{ 13 (1, 1, 1), 12 (1, 1, 0), 16 (1, 1, 2)} is orthonormal

271

2. In C [0, 2] with the inner product

hf , g i =

f (x)g (x)dx
0

The set {sin(x), cos(x)} is orthogonal but not orthonormal.

The set { 1 sin(x), 1 cos(x)} is orthonormal
The (infinite) set
1
1
1
1
1
{ , sin(x), cos(x), sin(2x), cos(2x), . . . }

2
is orthonormal.

272

Orthonormal bases
Bases that are orthonormal are particularly convenient to work with.
For example, we have the following

Lemma
If {v1 , . . . , vn } is an orthonormal basis for V and x V , then
x = hx, v1 iv1 + + hx, vn ivn
Proof: Exercise!

273

Orthogonal projection

[AR 6.2]

Let V be a real vector space, with inner product h, i

Let u V be a unit vector (i.e., kuk = 1)

Definition
The orthogonal projection of v onto u is
p = hv, uiu
Note

p = kvk cos u

v p is orthogonal to u

Example
The orthogonal projection (using the dot product) of (2, 3) onto
1 (1, 1) is:
2

274

More generally, we can project onto a subspace W of V as follows.

Let {u1 , . . . , uk } be an orthonormal basis for W .

Definition
The orthogonal projection of v V onto the subspace W is:
projW (v) = hv, u1 iu1 + + hv, uk iuk
Properties of p = projW (v)
pW

if w W , then projW (w) = w (by the above lemma)

v p is orthogonal to W (i.e., orthogonal to every vector in W )

p is the vector in W that is closest to v

That is, for all w W , kv wk > kv pk

p does not depend on the choice of orthonormal basis for W

275

Example
Let W = {(x, y , z) | x + y + z = 0} in V = R3 with the dot product.
The set
1
1
{u1 = (1, 1, 0), u2 = (1, 1, 2)}
2
6
is an orthonormal basis for W .

For v = (1, 2, 3) we have

projW (v) = hv, u1 iu1 + hv, u2 iu2
=
=
Note that v projW (v) =

is orthogonal to W .

276

6.5 Gram-Schmidt orthogonalization procedure

The following procedure can be used to make any basis orthonormal:

Gram-Schmidt procedure
Suppose {v1 , . . . , vk } is a basis for V .
1
kv1 k v1

1.

Let u1 =

2a.

w2 = v2 hv2 , u1 iu1

2b.

u2 =

3a.

w3 = v3 hv3 , u1 iu1 hv3 , u2 iu2

1
kw2 k w2

3b.
..
.
ka.

u3 = kw13 k w3
..
.
wk = vk hvk , u1 iu1 hvk , uk1 iuk1

kb.

uk =

1
kwk k wk

Then {u1 , . . . , uk } is an orthonormal basis for V .

277

Example
Find an orthonormal basis for the subspace W of R4 (with dot product)
spanned by
{(1, 1, 1, 1), (2, 4, 2, 4), (1, 5, 1, 3)}
Answer

1
1
1
{ (1, 1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1)}
2
2
2

Example
Find the point in W closest to v = (2, 2, 1, 3)
Answer
1
2 (3, 5, 3, 5)

278

279

6.6 Application: Curve Fitting

(AR 6.4)

Problem: Given a set of data points (x1 , y1 ), (x2 , y2 ),. . . , (xn , yn ) we

want to find the straight line y = a + bx which best approximates the
data.
A common approach is to minimise the least squares error , E
y

E 2 = sum of the squares

of vertical errors
n
X
i2
=

a+bx

i =1

n
X
(yi (a + bxi ))2
=
i =1

280

So given (x1 , y1 ),. . . , (xn , yn ) we want to find a, b R which minimise

n
X
(yi (a + bxi ))2
i =1

This can be written as

ky Auk2
where

y1
..
y=.
yn

1 x1
1 x2

A = . .
.
.
. .
1 xn

 
a
u=
b

To minimise ky Auk we want Au to be as close as possible to y

We can use projection to find the closest point.

281

We seek the vector, Au, in

W = {Av | v R2 }

(= the column space of A)

that is closest to y
The closest vector is precisely projW y
So to find u we could project y to W to get Au = projW y
and from this calculate u

282

However, we can calculate u directly (without finding an orthonormal

basis for W ) by noting that
hw, y projW yi = 0

for all w W

= hAv, y Aui = 0 for all v R2

= (Av)T (y Au) = 0 for all v R2
= vT AT (y Au) = 0 for all v R2
= AT (y Au) = 0
= AT y AT Au = 0
= AT Au = AT y
From this we can calculate u, given that we know A and y.
283

Summary
Given data points (x1 , y1 ), (x2 , y2 ),. . . , (xn , yn ), to find the straight line
y = a + bx of
 best
 fit,
a
we find u =
satisfying the normal equation
b
AT Au = AT y

()

1 x1
y1
1 x2
..

where y = . and A = .. ..
. .
yn
1 xn

If AT A is invertible (and it usually is), the solution to () is given by

u = (AT A)1 AT y

284

Example
Find the straight line which best fits the data points
(1, 1), (1, 1), (2, 3)

Answer
Line of best fit is: y =

9
7

+ 47 x

285

Extension
The same method works for finding quadratic fitting curves.
To find the quadratic y = a + bx + cx 2 which best fits data (x1 , y1 ),
(x2 , y2 ),. . . , (xn , yn ) we take

1 x1 x12
y1
1 x2 x 2
2

..
A = . .
y=.
..
.. ..
.
yn
1 xn xn2
and solve

AT Au = AT y
for

a
u = b
c
286