Machine Learning (CS 567) Lecture 5

Fall 2008

Time: T-Th 5:00pm - 6:20pm

Location: GFS 118

Instructor: Sofus A. Macskassy (macskass@usc.edu)

Office: SAL 216
Office hours: by appointment

Teaching assistant: Cheol Han (cheolhan@usc.edu)

Office: SAL 229
Office hours: M 2-3pm, W 11-12

Class web page:

http://www-scf.usc.edu/~csci567/index.html
Fall 2008 1 Lecture 5 - Sofus A. Macskassy
 Administrative: Regrading
• This regrading policy holds for quizzes, homeworks,
midterm and final exam.

• Grading complaint policy: if students have problems at

assignment grading, feel free to talk to the instructor/TA for
it within one week of getting the graded work back. After
one week no regrading will be done.

• Even if students only request re-grading one of the

answers, all this assignment will be checked and graded
again. (take the risk of possible losing total points.)

Fall 2008 2 Lecture 4 - Sofus A. Macskassy

 Lecture 5 Outline

• Logistic Regression (from last week)

• Linear Discriminant Analysis
• Off-the-shelf Classifiers

Fall 2008 3 Lecture 5 - Sofus A. Macskassy

 Linear Threshold Units
(
+1 if w1x1 + : : : + wnxn ¸ w0
h(x) =
¡1 otherwise

• We assume that each feature xj and each weight

wj is a real number (we will relax this later)
• We will study three different algorithms for
learning linear threshold units:
– Perceptron: classifier
– Logistic Regression: conditional distribution
– Linear Discriminant Analysis: joint distribution

Fall 2008 4 Lecture 5 - Sofus A. Macskassy

 Linear Discriminant Analysis
• Learn P(x,y). This is sometimes
called the generative approach,
because we can think of P(x,y) as a
model of how the data is generated.
y
– For example, if we factor the joint
distribution into the form
P(x,y) = P(y) P(x | y)
– we can think of P(y) as “generating” a
value for y according to P(y). Then we
can think of P(x | y) as generating a x
value for x given the previously-
generated value for y.
– This can be described as a Bayesian
network

Fall 2008 5 Lecture 5 - Sofus A. Macskassy

 Linear Discriminant Analysis (2)
• P(y) is a discrete multinomial distribution
– example: P(y = 0) = 0.31, P(y = 1) = 0.69 will
y

generate 31% negative examples and 69%

positive examples
• For LDA, we assume that P(x | y) is a
multivariate normal distribution with mean x

k and covariance matrix 

µ ¶
1 1
P (xjy = k) = exp ¡ [x ¡ ¹k ]T §¡1[x ¡ ¹k ]
(2¼)n=2j§j1=2 2

Fall 2008 6 Lecture 5 - Sofus A. Macskassy

 Multivariate Normal Distributions:
A tutorial
• Recall that the univariate normal (Gaussian) distribution has the formula
" #
1 1 (x ¡ ¹) 2
p(x) = exp ¡
(2¼)1=2¾ 2 ¾2
• where  is the mean and 2 is the variance
• Graphically, it looks like this:

Fall 2008 7 Lecture 5 - Sofus A. Macskassy

 The Multivariate Gaussian
• A 2-dimensional Gaussian is defined by a
mean vector  = (1,2) and a covariance
matrix " 2 #
2
¾1;1 ¾1;2
§= 2 2
¾1;2 ¾2;2

• where 2i,j = E[(xi – i)(xj - j)] is the

variance (if i = j) or co-variance (if i  j).

Fall 2008 8 Lecture 5 - Sofus A. Macskassy

 The Multivariate Gaussian (2)
" #
• If  is the identity matrix § = 1 0 and
0 1
 = (0, 0), we get the standard normal
distribution:

Fall 2008 9 Lecture 5 - Sofus A. Macskassy

 The Multivariate Gaussian (3)
• If  is a diagonal matrix, then x1, and x2 are independent random
variables, and lines of equal probability are ellipses parallel to the
coordinate "
axes. # For example, when
2 0
§= and
0 1

¹ = (2; 3) we obtain

Fall 2008 10 Lecture 5 - Sofus A. Macskassy

 The Multivariate Gaussian (4)
• Finally, if  is an arbitrary matrix, then x1 and x2 are
dependent, and lines of equal probability are ellipses tilted
relative to the coordinate axes. For example, when
" #
2 0:5
§= and
0:5 1
¹ = (2; 3) we obtain

Fall 2008 11 Lecture 5 - Sofus A. Macskassy

 Estimating a Multivariate Gaussian
• Given a set of N data points {x1, …, xN}, we can
compute the maximum likelihood estimate for the
multivariate Gaussian distribution as follows:
1 X
^ =
¹ xi
N i
1 X
^ =
§ (xi ¡ ¹ ^)T
^) ¢ (xi ¡ ¹
N i

• Note that the dot product in the second equation is an

outer product. The outer product of two vectors is a
matrix: 2 3 2 3
x1 x1y1 x1y2 x1y3
x¢yT = 4 x2 5¢[y1 y2 y3] = 4 x2y1 x2y2 x2y3 7
6 7 6
5
x3 x3y1 x3y2 x3y3

• For comparison, the usual dot product is written as xT¢ y

Fall 2008 12 Lecture 5 - Sofus A. Macskassy
 The LDA Model
• Linear discriminant analysis assumes that the joint
distribution has the form
P(x|y)
µ ¶
1 1 T ¡1
P (x; y) = P (y) n=2 1=2
exp ¡ [x ¡ ¹y ] § [x ¡ ¹y ]
(2¼) j§j 2

where each y is the mean of a multivariate

Gaussian for examples belonging to class y and
 is a single covariance matrix shared by all
classes.

Fall 2008 13 Lecture 5 - Sofus A. Macskassy

 Fitting the LDA Model
• It is easy to learn the LDA model in two passes through
the data:
– Let ¼^k be our estimate of P(y = k)
– Let Nk be the number of training examples belonging to class k.
Nk
^k =
¼
N
1 X
¹^k = xi
Nk fi:y =kg
i

1X
^ =
§ (xi ¡ ¹ ^yi )T
^yi ) ¢ (xi ¡ ¹
N i
• Note that each xi is subtracted from its corresponding ¹
^yi
prior to taking the outer product. This gives us the
“pooled” estimate of 
Fall 2008 14 Lecture 5 - Sofus A. Macskassy
 LDA learns an LTU
• Consider the 2-class case with a 0/1 loss function. Recall that
P (x; y = 0)
P (y = 0jx) =
P (x; y = 0) + P (x; y = 1)
P (x; y = 1)
P (y = 1jx) =
P (x; y = 0) + P (x; y = 1)
• Also recall from our derivation of the Logistic Regression classifier
that we should classify into class ŷ = 1 if
P (y = 1jx)
log > 0
P (y = 0jx)
• Hence, for LDA, we should classify into ŷ = 1 if
P (x; y = 1)
log > 0
P (x; y = 0)
because the denominators cancel

Fall 2008 15 Lecture 5 - Sofus A. Macskassy

 LDA learns an LTU (2)
µ ¶
1 1 T §¡1[x ¡ ¹ ]
P (x; y) = P (y) exp ¡ [x ¡ ¹y ] y
(2¼)n=2j§j1=2 2
³ ´
P (y = 1) 1 1 T ¡1
exp ¡ 2 [x ¡ ¹1] § [x ¡ ¹1]
P (x; y = 1) (2¼)n=2j§j1=2
= ³ ´
P (x; y = 0) P (y = 0) 1 1 T ¡1
exp ¡ 2 [x ¡ ¹0] § [x ¡ ¹0]
(2¼)n=2j§j1=2
³ ´
P (y = 1 T ¡1
1) exp ¡ 2 [x ¡ ¹1] § [x ¡ ¹1]
P (x; y = 1)
= ³
1
´
P (x; y = 0) P (y = T ¡1
0) exp ¡ 2 [x ¡ ¹0] § [x ¡ ¹0]

P (x; y = 1) P (y = 1) 1 ³ T ¡1 T ¡1
´
log = log ¡ [x ¡ ¹1] § [x ¡ ¹1] ¡ [x ¡ ¹0] § [x ¡ ¹0]
P (x; y = 0) P (y = 0) 2

Fall 2008 16 Lecture 5 - Sofus A. Macskassy

 LDA learns an LTU (3)
• Let’s focus on the term in brackets:
³ ´
[x ¡ ¹1]T §¡1[x ¡ ¹1] ¡ [x ¡ ¹0]T §¡1[x ¡ ¹0]

• Expand the quadratic forms as follows:

[x ¡ ¹1]T §¡1[x ¡ ¹1] = xT §¡1x ¡ xT §¡1¹1 ¡ ¹T ¡1 T ¡1
1 § x + ¹1 § ¹1

[x ¡ ¹0]T §¡1[x ¡ ¹0] = xT §¡1x ¡ xT §¡1¹0 ¡ ¹T

0 §¡1
x + ¹T ¡1
0 § ¹0

• Subtract the lower from the upper line and collect similar
terms. Note that the quadratic terms cancel! This
leaves only terms linear in x.
xT §¡1(¹0¡¹1)+(¹0¡¹1)§¡1x+¹T
1 §¡1¹ ¡¹T §¡1¹
1 0 0

Fall 2008 17 Lecture 5 - Sofus A. Macskassy

 LDA learns an LTU (4)
xT §¡1(¹0¡¹1)+(¹0¡¹1)§¡1x+¹T
1 §¡1¹ ¡¹T §¡1¹
1 0 0

• Note that since -1 is symmetric aT §¡1b = bT §¡1a

for any two vectors a and b. Hence, the first two terms
can be combined to give
2xT §¡1(¹0 ¡ ¹1) + ¹T 1 § ¡1¹ ¡ ¹T §¡1¹ :
1 0 0
• Now plug this back in…
P (x; y = 1) P (y = 1) 1 h T ¡1 T ¡1 T ¡1
i
log = log ¡ 2x § (¹0 ¡ ¹1) + ¹1 § ¹1 ¡ ¹0 § ¹0
P (x; y = 0) P (y = 0) 2
P (x; y = 1) P (y = 1) 1 ¡1¹ + 1 ¹T §¡1¹
log = log + xT §¡1(¹1 ¡ ¹0) ¡ ¹T § 1 0
P (x; y = 0) P (y = 0) 2 1 2 0

Fall 2008 18 Lecture 5 - Sofus A. Macskassy

 LDA learns an LTU (5)
P (x; y = 1) P (y = 1) 1 ¡1¹ + 1 ¹T §¡1¹
log = log + xT §¡1(¹1 ¡ ¹0) ¡ ¹T § 1 0
P (x; y = 0) P (y = 0) 2 1 2 0

Let

w = §¡1(¹1 ¡ ¹0)
P (y = 1) 1 T ¡1 1 T ¡1
c = log ¡ ¹1 § ¹1 + ¹0 § ¹0
P (y = 0) 2 2

Then we will classify into class y^ = 1 if

w ¢ x + c > 0:
This is an LTU.
Fall 2008 19 Lecture 5 - Sofus A. Macskassy
 Two Geometric Views of LDA
View 1: Mahalanobis Distance
2 T ¡1
• The quantity DM (x; u) = (x ¡ u) § (x ¡ u) is known as
the (squared) Mahalanobis distance between x and u. We can think
of the matrix -1 as a linear distortion of the coordinate system that
converts the standard Euclidean distance into the Mahalanobis
distance
• Note that
1
log P (xjy = k) / log ¼k ¡ [(x ¡ ¹k )T §¡1(x ¡ ¹k )]
2
1
log P (xjy = k) / log ¼k ¡ DM (x; ¹k )2
2
• Therefore, we can view LDA as computing
– DM (x; ¹0) and DM (x; ¹1)2
2
and then classifying x according to which mean 0 or 1 is closest in
Mahalanobis distance (corrected by log k)
Fall 2008 20 Lecture 5 - Sofus A. Macskassy
 Mahalanobis Distance
• Which point is closer to the vertical line?

• Weak point of mahalanobis: if only few points

observed, then the variance is not meaningful.

Fall 2008 21 Lecture 5 - Sofus A. Macskassy

 View 2: Most Informative Low-
Dimensional Projection
• LDA can also be viewed as finding a hyperplane of
dimension K – 1 such that x and the {k} are projected
down into this hyperplane and then x is classified to the
nearest k using Euclidean distance inside this hyperplane

Fall 2008 22 Lecture 5 - Sofus A. Macskassy

 Nonlinear boundaries with LDA

Left: Linear decision boundary found by LDA

Right: Quadratic decision boundary found by LDA. This was

obtained by finding linear decision boundaries in the five
dimensional space X1, X2, X1*X2, X12, X22. Linear
inequalities in this space are quadratic inequalities in
the original space.
Fall 2008 23 Lecture 5 - Sofus A. Macskassy
 Generalizations of LDA
• General Gaussian Classifier
– Instead of assuming that all classes share the same , we can
allow each class k to have its own k. In this case, the resulting
classifier will be a quadratic threshold unit (instead of an LTU)
• Naïve Gaussian Classifier
– Allow each class to have its own k, but require that each k be
diagonal. This means that within each class, any pair of features
xj1 and xj2 will be assumed to be statistically independent. The
resulting classifier is still a quadratic threshold unit (but with a
restricted form)

Fall 2008 24 Lecture 5 - Sofus A. Macskassy

Quadratic Discriminant Analysis (QDA)
Quadratic Discriminant Analysis (QDA) is a generalization of LDA where
different classes are allowed to have different covariance matrices. i.e.,
k  l for classes k  l. ^
§k =
1 X
(xi ¡ ¹^k ) ¢ (xi ¡ ¹^k )T
Nk i;y =k
i
µ ¶
1 1 T §¡1[x ¡ ¹ ]
P (x; y = k) = P (y = k) exp ¡ [x ¡ ¹k ] k k
(2¼)n=2 j§k j1=2 2
We can solve for
P (y = 1jx)
log > 0
P (y = 0jx)
in a similar way as before. Note, however, that the 0 and 1
determinants do not cancel.

Fall 2008 25 Lecture 5 - Sofus A. Macskassy

 LDA vs. QDA

Left: Quadratic fit by LDA in transformed space.

Right: Quadratic fit by QDA.

The differences in the fit are very small.

Fall 2008 26 Lecture 5 - Sofus A. Macskassy

 Generalizations of LDA
• General Gaussian Classifier
– Instead of assuming that all classes share the same , we can
allow each class k to have its own k. In this case, the resulting
classifier will be a quadratic threshold unit (instead of an LTU)
• Naïve Gaussian Classifier
– Allow each class to have its own k, but require that each k be
diagonal. This means that within each class, any pair of features
xj1 and xj2 will be assumed to be statistically independent. The
resulting classifier is still a quadratic threshold unit (but with a
restricted form)

Fall 2008 27 Lecture 5 - Sofus A. Macskassy

 Different Discriminant Functions

Fall 2008 28 Lecture 5 - Sofus A. Macskassy

 Summary of
Linear Discriminant Analysis

• Learns the joint probability distribution P(x, y).

• Direct Computation. The maximum likelihood estimate
of P(x,y) can be computed from the data without
search. However, inverting the  matrix requires O(n3)
time.
• Eager. The classifier is constructed from the training
examples. The examples can then be discarded.
• Batch. Only a batch algorithm is available. An online
algorithm could be constructed if there is an online
algorithm for incrementally updated -1. [This is easy
for the case where  is diagonal.]

Fall 2008 29 Lecture 5 - Sofus A. Macskassy

 Comparing Perceptron, Logistic
Regression, and LDA
• How should we choose among these three
algorithms?
• There is a big debate within the machine
learning community!

Fall 2008 30 Lecture 5 - Sofus A. Macskassy

 Issues in the Debate

• Statistical Efficiency. If the generative model

P(x,y) is correct, then LDA usually gives the
highest accuracy, particularly when the amount
of training data is small. If the model is correct,
LDA requires 30% less data than Logistic
Regression in theory
• Computational Efficiency. Generative models
typically are the easiest to learn. In our
example, LDA can be computed directly from the
data without using gradient descent.

Fall 2008 31 Lecture 5 - Sofus A. Macskassy

 Issues in the Debate
• Robustness to changing loss functions. Both generative
and conditional probability models allow the loss function
to be changed at run time without re-learning.
Perceptron requires re-training the classifier when the
loss function changes.
• Robustness to model assumptions. The generative
model usually performs poorly when the assumptions
are violated. For example, if P(x | y) is very non-
Gaussian, then LDA won’t work well. Logistic Regression
is more robust to model assumptions, and Perceptron is
even more robust.
• Robustness to missing values and noise. In many
applications, some of the features xij may be missing or
corrupted in some of the training examples. Generative
models typically provide better ways of handling this
than non-generative models.
Fall 2008 32 Lecture 5 - Sofus A. Macskassy