Anda di halaman 1dari 38

# Statistical Learning and Data Mining

## CS 363D/ SSC 358

Lecture: Linear Algebra Foundations

Outline
Vectors (Norms, Distances, Inner Products, Orthogonality, Linear
Combinations, Linear Independence, Linear Subspace, Basis, Orthogonal
Basis)

Vectors

## Can be imbue such vectors with properties possessed by real numbers

(also called scalars)?

Vectors
v
u
u-v

u
u+v

v
v

Vectors
v
u
u-v

u
u+v

v
v

## Can add and subtract vectors

Commutative: u + v = v + u

Vectors
v
u
u-v

u
u+v

v
v

## Can add and subtract vectors

Commutative: u + v = v + u
Associative: u + (v + w) = (u + v) + w

Vectors
v
u
u-v

u
u+v

v
v

## Can add and subtract vectors

Commutative: u + v = v + u
Associative: u + (v + w) = (u + v) + w
Zero: There exists a vector 0, such that u + 0 = u

Vectors
v
u
u-v

u
u+v

v
v

## Can add and subtract vectors

Commutative: u + v = v + u
Associative: u + (v + w) = (u + v) + w
Zero: There exists a vector 0, such that u + 0 = u
Inverse: For every u, there is a vector - u, such that u + (-u) = 0

Vectors
u

2u

Vectors
u

2u

## Can multiply vectors with scalars

Associative: a (b u) = (ab) u

Vectors
u

2u

## Can multiply vectors with scalars

Associative: a (b u) = (ab) u
Distributive I: (a + b) u = a u + b u

Vectors
u

2u

## Can multiply vectors with scalars

Associative: a (b u) = (ab) u
Distributive I: (a + b) u = a u + b u
Distributive II: a (u + v) = a u + a v

Vectors
u

2u

## Can multiply vectors with scalars

Associative: a (b u) = (ab) u
Distributive I: (a + b) u = a u + b u
Distributive II: a (u + v) = a u + a v
Identity: 1 u = u

Vector Space
A vector space is a set of vectors, along with associated scalars (typically:
real numbers), that satisfy properties in previous two slides, and that are
closed under vector addition and scalar multiplication

## not just in data mining/machine learning but in many applications across

science and engineering

And from the previous two slides, we can treat them like ordinary numbers
for the most part

## Vector Space: Linear Independence

Suppose we have three vectors x1 , x2 , and x3 , and that x1 =
Then x1 is linearly dependent on x2 and x3 .

2 x2 +

## When are x1 , x2 , . . . , xn linearly independent?

x1 , x2 , . . . , xn are linearly independent
If 1 x1 + 2 x2 + . . . + n xn = 0, then 1 =

= ... =

= 0.

3 x3 .

## Vector Space: Linear Independence

Suppose we have three vectors x1 , x2 , and x3 , and that x1 =
Then x1 is linearly dependent on x2 and x3 .

2 x2 +

## When are x1 , x2 , . . . , xn linearly independent?

x1 , x2 , . . . , xn are linearly independent
If 1 x1 + 2 x2 + . . . + n xn = 0, then 1 =

= ... =

= 0.

3 x3 .

## Vector Space: Linear Independence

Suppose we have three vectors x1 , x2 , and x3 , and that x1 =
Then x1 is linearly dependent on x2 and x3 .

2 x2 +

## When are x1 , x2 , . . . , xn linearly independent?

x1 , x2 , . . . , xn are linearly independent
If 1 x1 + 2 x2 + . . . + n xn = 0, then 1 =

= ... =

= 0.

3 x3 .

## Vector Space: Subspace

A linear subspace is a set of vectors that is closed under vector addition
and scalar multiplication: if x1 and x2 belong to the subspace, then so do
1 x1 + 2 x2 .
A basis of the subspace is the maximal set of vectors in the subspace that
are linearly independent of each other.
An orthogonal basis is a basis where all basis vectors are orthogonal to
each other.

## Vector Space: Subspace

A linear subspace is a set of vectors that is closed under vector addition
and scalar multiplication: if x1 and x2 belong to the subspace, then so do
1 x1 + 2 x2 .
A basis of the subspace is the maximal set of vectors in the subspace that
are linearly independent of each other.
An orthogonal basis is a basis where all basis vectors are orthogonal to
each other.

Vectors
x=
2

1
2

=1

1
0

+2

0
1

## A vector space is thus the set of vectors obtained as linear combinations of

its basis vectors

Can thus represent a vector as an array of numbers: where the numbers are
the coecients of the basis vectors in the linear combination

Vectors
A vector can be thought of as having a direction and a magnitude

Vectors
A vector can be thought of as having a direction and a magnitude
This magnitude is called a vector norm

Vectors
A vector can be thought of as having a direction and a magnitude
This magnitude is called a vector norm
Properties satisfied by vector norms || . ||

Vectors
A vector can be thought of as having a direction and a magnitude
This magnitude is called a vector norm
Properties satisfied by vector norms || . ||
|| x || >= 0 and || x || = 0 if and only if x = 0

Vectors
A vector can be thought of as having a direction and a magnitude
This magnitude is called a vector norm
Properties satisfied by vector norms || . ||
|| x || >= 0 and || x || = 0 if and only if x = 0
|| a x || = | a | || x ||

(Homegeneity)

Vectors
A vector can be thought of as having a direction and a magnitude
This magnitude is called a vector norm
Properties satisfied by vector norms || . ||
|| x || >= 0 and || x || = 0 if and only if x = 0
|| a x || = | a | || x ||

(Homegeneity)

|| x + y || <= || x || + || y ||

(Triangle Inequality)

2

x1
x2
..
.

6
6
x=6
4
x2 =

xn

3
7
7
7
5

xp =

p
p

: 1-norm

x1 = max |xi |
i=1

-norm

: p-norm

Distances
x=

1
2

y=

2
1

## How do we measure the distance between two vectors?

We looked at a few distance measures in the previous class; which could
be looked at as distances between vectors
One could also use vector norms to compute distances:

1
2
2
2
x y2 =
= (1 2) + (2 1) = 2
2
1
2
x

y1 = 2

y1 = 1

Metrics
A distance d(x, y) is a metric i
d(x, y)

0, and d(x, y) = 0 i x = y

d(x, y) = d(y, x)

(Symmetry)

## d(x, z) d(x, y) + d(y, z)

(Triangle Inequality)

Metrics
A distance d(x, y) is a metric i
d(x, y)

0, and d(x, y) = 0 i x = y

d(x, y) = d(y, x)

(Symmetry)

## d(x, z) d(x, y) + d(y, z)

(Triangle Inequality)

d(x, z) = kx
X

zk = k(x

d(x, y) = kx

y) + (y

## yk. Is this a valid metric?

z)k kx

yk is a valid metric.

yk + ky

## zk = d(x, y) + d(y, z).

Metrics
A distance d(x, y) is a metric i
d(x, y)

0, and d(x, y) = 0 i x = y

d(x, y) = d(y, x)

(Symmetry)

## d(x, z) d(x, y) + d(y, z)

(Triangle Inequality)

d(x, z) = kx
X

zk = k(x

d(x, y) = kx

y) + (y

## yk. Is this a valid metric?

z)k kx

yk is a valid metric.

yk + ky

## zk = d(x, y) + d(y, z).

Metrics
A distance d(x, y) is a metric i
d(x, y)

0, and d(x, y) = 0 i x = y

d(x, y) = d(y, x)

(Symmetry)

## d(x, z) d(x, y) + d(y, z)

(Triangle Inequality)

d(x, z) = kx
X

zk = k(x

d(x, y) = kx

y) + (y

## yk. Is this a valid metric?

z)k kx

yk is a valid metric.

yk + ky

## Inner Products (Also: Dot Products)

2

6
6
x=6
4

x1
x2
..
.
xn

7
7
7
5

6
6
y=6
4

y1
y2
..
.
yn

3
7
7
7
5

Inner Product: x y = x1 y1 + x2 y2 + . . . + xn yn =
2
3
y1
7
y
6

2
7
6
Can be viewed as: x1 x2 . . . xn 6 . 7
4 .. 5
yn
Examples: x2 = kxk22 , kx

yk22 = (x

y)T (x

y)

Pn

i=1

x i yi

## Inner Products (Also: Dot Products)

2

6
6
x=6
4

x1
x2
..
.
xn

7
7
7
5

6
6
y=6
4

y1
y2
..
.
yn

3
7
7
7
5

Inner Product: x y = x1 y1 + x2 y2 + . . . + xn yn =
2
3
y1
7
y
6

2
7
6
Can be viewed as: x1 x2 . . . xn 6 . 7
4 .. 5
yn
Examples: xT x = kxk22 , (x

y)T (x

y) = kx

Pn

yk22

i=1

x i yi

Projections
x=

1
2

y=

2
1

xT y
cos =
kxk2 kyk2

Projections
x=

1
2

y=

2
1

## xT y = kxk2 kyk2 cos

xT y
cos =
kxk2 kyk2

Projection of x onto y:
Magnitude: kxk2 cos = xT

y
kyk2

= xT

yb
|{z}

Unit norm

Orthogonal
y

## Vector Space: Subspace

A linear subspace is a set of vectors that is closed under vector addition
and scalar multiplication: if x1 and x2 belong to the subspace, then so do
1 x1 + 2 x2 .
A basis of the subspace is the maximal set of vectors in the subspace that
are linearly independent of each other.
An orthogonal basis is a basis where all basis vectors are orthogonal to
each other.