CSE/MATH 6643: Numerical Linear Algebra

Haesun Park
hpark@cc.gatech.edu

School of Computational Science and Engineering

College of Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA

Lecture 14

Least Square Problem SVD

7.4.4

Solving LS

min kAx bk2 , A Rmn , b Rm1 , m n.

x

Let the SVD of A be

A = U V

U1

U2

"

1
0

0
0

V1

V2

where U1 Rmr , U2 Rm(mr) , 1 = diag(1 , , r ), 1 r > 0,

and V1 Rnr , V2 Rn(nr) ,i.e. rank(A) = r.

CSE/MATH 6643: Numerical Linear Algebra p.1/12

Least Square Problem SVD

kAx bk2

=
=

T
T
U V x b = U (U V x b)
2
2

T
T
V x U b
2

8
>
>
<
>
>
:

V1T
V2T

Letting V T x =
T

U b=

"

1

0
0

U1T
U2T

#"

y
z

b=

"

x=

nr

mr

#
# "

c 1 y c

d
d

CSE/MATH 6643: Numerical Linear Algebra p.2/12

Least Square Problem SVD

Since 1 Rrr , non-singular, we can find the unique solution for
1 y c = 0 1 y = c y = 1
1 c.
Letting r(x) = kAx bk2 , the residual r(xLS ) = kdk where xLS is the LS
solution.
The solution is
xLS = V

"

y
z

where y = 1
1 c and z can be anything. (if rank(A) = n, z is null).
!

y
= V1 y + V2 z
xLS = V1 V2
z
Note V2 z null(A).

When z = 0, xLS = V

"

1
1 c
0

is called minimum-norm solution.

CSE/MATH 6643: Numerical Linear Algebra p.3/12

Least Square Problem Rank Decision

7.4.5

Example

A R64 , 4 (A) = 1016 , 3 (A) = 1014 .

2
1
6
0.5
6
6
6
1014
6
T
A = U V = U 6
6
1016
6
6
4

7
7
7
7
7 T
7V
7
7
7
5

Depending on rank decision (2 or 3 or 4?), we obtain very different solutions.

If we consider
the tolerance 3s.t. 1016
32
2 < , and rank(A)
2
1
1
76
6
7
6
76
7,y = 6
1 = 6
2
0.5
54
4
5
4
14
14
10
10

= 3,
3
x
7
x 7
5.
x

If we consider the tolerance s.t. 1014 < , and rank(A) = 2,

"
#
"
#"
#
1
1
x
1 =
,y =
.
0.5
2
x

CSE/MATH 6643: Numerical Linear Algebra p.4/12

Least Square Problem Rank Decision

Min-norm solution:
the first case, xLS

2 2

6 6
6 4
=V 6
4

the second case, xLS

2 "

6
=V 4

1014
0
#"

32

3 3

x
76
7 7
54 x 5 7
7
5
x
x
x

# 3
7
5

0
Determine numerical rank can be difficult. Usually we find a large gap.
2
3
1
6
7
101
6
7
6
7
102
6
7 no gap?
6
7
3
6
7
10
4
5
..
.

CSE/MATH 6643: Numerical Linear Algebra p.5/12

Least Square Problem

In some applications, just need to compute the size of the residual vector.
= Can be done WITHOUT computing the solution vector x.
r = AxLS b
krk2 = kAxLS bk2
r

=
=
=
=

krk2 = kU

0
d

U V T xLS b
!
!
1 0
y
U
U
0 0
z
!
1 y c
U
d
!
0
U
d

k2 = kdk2 .
CSE/MATH 6643: Numerical Linear Algebra p.6/12

c
d

Least Square Problem

Full Rank case
h

Theorem. If A Rmn , rank(A) = n, A = a1 an ,

i
i h
h
Q = q1 qm = Q1 Q2 where Q1 Rnn , Q2 R(mn)n ,
"
#
R
A=Q
,
0
then
span {a1 , , ak } = span {q1 , , qk } (1 k n)

CSE/MATH 6643: Numerical Linear Algebra p.7/12

Least Square Problem Rank Deficient

Proof: A = Q

a1

ak =

k
P

"

an

R
0
i

#
h

q1

Q1

Q2

qn

"
2

i6
6
4

r11

= Q1 R

..
.

r1n
.. 7
7
. 5
rnm

qi rik span {q1 , , qk } = span {a1 , , ak } span {q1 , , qk }

i=1

As the dimensions of both span {a1 , , ak } and span {q1 , , qk } are k,

span {a1 , , ak } = span {q1 , , qk }.
Range(A) = R(A) = span {a1 , , ak } = span {q1 , , qk } = R(Q1 )

m1 T
R (A) = z R
|z w = 0 for w R(A)

CSE/MATH 6643: Numerical Linear Algebra p.8/12

Least Square Problem Rank Deficient

QRD with Column Pivoting for Deficient Rank
If A = QR and rank(A)
i 1 , , qk },
i {a1 , h , ak } = span {q
h = n, then span

1 k n where A =

a1

an

,Q=

q1

qn

Why QRD with Column Pivoting?

2

6
Eg. A = 6
4

Consider QRD of A

7
1 7
5.rank(A) = 2.
1
2

6
A = QR = 6
4

32

1
1
1

76
76
54

7
1 7
5
1

Although rank(A) = 2, we dont have Range(A) = span {qi , qj : i 6= j}. QRD with
C.P. can help us to maintain span {a1 , , ak } = span {q1 , , qk } in rank deficient
case.

CSE/MATH 6643: Numerical Linear Algebra p.9/12

Least Square Problem Rank Deficient

For any A Rmn , QRD with Column Pivoting computes
!
R
A = Q
0
, where
0

B
R=@

R11
0
|{z}

(n r) r

R12
0
|{z}

(n r) (n r)

1
C
A

where R11 Rrr is upper-triangular matrix, R12 Rrnr ,

r = rank(A) = rank(R) = rank(R11 ).
Q Rmm , orthogonal; Rnn , permutation.

CSE/MATH 6643: Numerical Linear Algebra p.10/12

Least Square Problem Rank Deficient

Solving LS
A = Q

"

R
0

6
= Q6
4

R11

T x =

y
z

"

11

T
x b =

R12

R12

nr

R11
6
6
kAx bk2 =
4 0

R12

Letting

"

R11

R12

"

R11

kAx bk2 = Q

, QT b =

R12
0
0

6
7
7 A = Q 6
4
5

"

We can choose

mr

"
# "
#

7
y
c
=
7

z
d

kAx bk22 = kR11 y + R12 z ck22 + kdk22

!

7 T
7 .
5
#

T
T
x Q b

,
!

R11 y + R12 z c

so that

z
R11 y + R12 z c = 0 R11 y = c R12 z

CSE/MATH 6643: Numerical Linear Algebra p.11/12

Least Square Problem Rank Deficient

z can be anything, and y can be chosen so that R11 y = c R12 z. Can set
z = 0, then y satisfies R11 y = c.
"
#
1
R11 (c R12 z)
x=
where z is free.
z
#
"
1
R11 c
.
If we set z = 0, we get basic solution x =
0
#
"
R11
and R11 = R.
When rank(A) = n, A = Q
0

CSE/MATH 6643: Numerical Linear Algebra p.12/12