Outline
Bayes Theorem
MAP Learners
Bayes optimal classifier
Nave Bayes classifier
Example text classification
Bayesian networks
EM algorithm
Bayesian Learning
Bayesian Learning
Bayes Theorem
If and are two random variables
()
=
()
()
(h) prior probability of hypothesis h
() prior probability of training data
h| probability of h given
|h probability of given h
Bayesian Learning
Example
Does the patient have cancer or not?
A patient takes a lab test and the result comes back
positive. The test returns a correct positive result in only
98% of the cases in which the disease is actually present,
and a correct negative result in only 97% of the cases in
which the disease is no present. Furthermore, 0.008 of
the entire population have this cancer.
Bayesian Learning
+| =
| =
+| =
| =
| + =
Bayesian Learning
Bayesian Learning
H
6
Then h =
Bayesian Learning
-(4)
-(P)
Bayesian Learning
10
H
6
Then
H
h =Q
Bayesian Learning
RS,T
, if h is consistent with
0 , otherwise
CSL465/603 - Machine Learning
11
P h|H
Bayesian Learning
P h|H, _
Evolution of Posterior
Probabilities
12
13
Example
hH = .4, hH = 0, + hH = 1
h_ = .3, h_ = 1, + h_ = 0
hc = .3, hc = 1, + hc = 0
f + h7 (h7 |) =
f h7 (h7 |) =
4= 6
4= 6
Bayesian Learning
14
15
Gibbs Classifier
Gibbs Algorithm
Choose a hypothesis h H at random, according to the
posterior probability distribution over
Use h to classify the new instance x.
Bayesian Learning
16
x ( )
x =
(x)
x - posterior probability that instance x belongs to
class
x - probability that an instance drawn from class
would be x (likelihood)
( ) probability of class (prior)
(x) probability of instance x (evidence)
Bayesian Learning
17
Bayesian Learning
18
Bayesian Learning
19
H , =
H
= argmax ( )
H
Bayesian Learning
20
Bayesian Learning
21
Bayesian Learning
22
Bayesian Learning
23
Bayesian Learning
24
Bayesian Learning
25
= 0
H
Bayesian Learning
26
27
|P |
=
|P|
|R d|
CSL465/603 - Machine Learning
28
Bayesian Learning
29
Example: 20 Newsgroup
Given 1000 training documents from each group
Learn to classify new documents according to the
newsgroup it came from
NBC 89% accuracy
Bayesian Learning
30
Bayesian Learning
31
BusTourGroup
S,B
Lightning
Campfire
0.4
0.1
0.8
0.2
0.6
0.9
0.2
0.8
Campfire
Thunder
Bayesian Learning
ForestFire
32
H, _, , = |
H
33
Learning
If the structure is known and there are no missing values,
it is easy to learn a Bayes network
If the network structure is known and there are some
missing values, expectation maximization algorithm
If the structure is unknown, the problem is very difficult.
Bayesian Learning
34
Summary
Bayes rule
Bayes Optimal Classifier
Practical Nave Bayes Classifier
Example text classification task
Maximum-likelihood estimates
Bayesian networks
Bayesian Learning
35