1

Page 1
Estimasi Estimasi Prob. Density Function Prob. Density Function
dengan dengan EM EM
Sumber Sumber: :
- -Forsyth & Ponce Chap. 7 Forsyth & Ponce Chap. 7
- -Standford Standford Vision & Modeling Vision & Modeling
Probability Density Estimation Probability Density Estimation
Parametric Representations
Non-Parametric Representations
Mixture Models
2
Page 2
Metode Metode estimasi estimasi Non Non- -parametric parametric
Tanpa asumsi apapun tentang distribusi
Estimasi sepenuhnya bergantung ada DATA
cara mudah menggunakan: Histogram
Histograms Histograms
Diskritisasi, lantas ubah dalam bentuk batang:
3
Page 3
Histograms Histograms
Butuh komputasi banyak, namun sangat umum
digunakan
Dapat diterapkan pada sembarang bentuk
densitas (arbitrary density)
Histograms Histograms
Permasalahan:
Higher dimensional Spaces:
- jumlah batang (bins) yg. Exponential
- jumlah training data yg exponential
- Curse of Dimensionality
size batang ? Terlalu sedikit: >> kasar
Terlalu banyak: >> terlalu halus
4
Page 4
Pendekatan Pendekatan secara secara prinsip prinsip: :
x diambil dari unknown p(x)
probabiliti bahwa x ada dalam region R adalah:
V x p dx x p P
R
) ( ' ) ' ( =

Pendekatan Pendekatan secara secara prinsip prinsip: :

V x p dx x p P
R
) ( ' ) ' ( =

N
K
P =
x diambil dari unknown p(x)
probabiliti bahwa x ada dalam region R adalah:
5
Page 5
Pendekatan Pendekatan secara secara prinsip prinsip: :
V x p dx x p P
R
) ( ' ) ' ( =

N
K
P =
NV
K
x p ) (
x diambil dari unknown p(x)
probabiliti bahwa x ada dalam region R adalah:
Pendekatan Pendekatan secara secara prinsip prinsip: :
NV
K
x p ) (
Dengan Fix V
Tentukan K
Dengan Fix K
Tentukan V
Metoda Kernel-Based
K-nearest
neighbor
6
Page 6
Metoda Metoda Kernel Kernel- -Based: Based:
NV
K
x p ) (
Parzen Window:

<
=
otherwise 0
2 / 1 | u | 1
) (
j
u H
Metoda Metoda Kernel Kernel- -Based: Based:
NV
K
x p ) (
Parzen Window:

<
=
otherwise 0
2 / 1 | u | 1
) (
j
u H

=
=
N
n
n
x x H K
1
) (
7
Page 7
Metoda Metoda Kernel Kernel- -Based: Based:
NV
K
x p ) (
Parzen Window:

<
=
otherwise 0
2 / 1 | u | 1
) (
j
u H

=
=
N
n
n
x x H K
1
) (

=
=
N
n
n
d
x x H
Nh
x p
1
) (
1
) (
Metoda Metoda Kernel Kernel- -Based: Based:
NV
K
x p ) (
Gaussian Window:

=

=
2
2
1
2 / 2
2
|| ||
exp
) 2 (
1 1
) (
h
x x
h N
x p
n
N
n
d

8
Page 8
Metoda Metoda Kernel Kernel- -Based: Based:
K K- -nearest nearest- -neighbor: neighbor:
NV
K
x p ) (
Kembankan V sampai dia mencapai K points.
9
Page 9
K K- -nearest nearest- -neighbor: neighbor:
K K- -nearest nearest- -neighbor: neighbor:
Klasifikasi secara Bayesian :
V N
K
C x p
k
k
k
= ) | (
NV
K
x p = ) (
N
N
C p
k
k
= ) (
10
Page 10
K K- -nearest nearest- -neighbor: neighbor:
Klasifikasi secara Bayesian :
V N
K
C x p
k
k
k
= ) | (
NV
K
x p = ) (
N
N
C p
k
k
= ) (
K
K
x C p
k
k
= ) | (
aturan klasifikasi k-nearest-neighbour
Probability Density Estimation Probability Density Estimation
Parametric Representations
Non-Parametric Representations
Mixture Models (Model Gabungan)
11
Page 11
Mixture Mixture- -Models (Model Models (Model Gabungan Gabungan): ):
Gaussians:
- Mudah
- Low Memory
- Cepat
- Good Properties
Non-Parametric:
- Umum
- Memory Intensive
- Slow
Mixture Models
Campuran Campuran fungsi fungsi Gaussian (mixture of Gaussian (mixture of
Gaussians): Gaussians):
x
p(x)
Jumlah dari Gaussians tunggal
12
Page 12
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
Jumlah dari Gaussians tunggal
Keunggulan: Dapat mendekati bentuk densitas
sembarang (Arbitrary Shape)
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
Generative Model: z
1
2
3
P(j)
p(x|j)
13
Page 13
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)

=
=
M
j
j P j x p x p
1
) ( ) | ( ) (

=
2
2
2 / 2
2
|| ||
exp
) 2 (
1
) | (
j
d
j
x
j x p

Campuran Campuran fungsi fungsi Gaussian: Gaussian:

Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
14
Page 14
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
0 =

k
E

E
k

Campuran Campuran fungsi fungsi Gaussian: Gaussian:

Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
0 =

k
E

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

15
Page 15
Campuran Campuran fungsi fungsi Gaussian: Gaussian:

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

Campuran Campuran fungsi fungsi Gaussian: Gaussian:

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

=
=
M
k
n
n
n
k P k x p
j P j x p
x j P
1
) ( ) | (
) ( ) | (
) | (
16
Page 16
Campuran Campuran fungsi fungsi Gaussian: Gaussian:

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

=
2
2
2 / 2
2
|| ||
exp
) 2 (
1
) | (
j
j n
d
j
n
x
j x p

=
=
M
k
n
n
n
k P k x p
j P j x p
x j P
1
) ( ) | (
) ( ) | (
) | (
Campuran Campuran fungsi fungsi Gaussian: Gaussian:

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

=
2
2
2 / 2
2
|| ||
exp
) 2 (
1
) | (
j
j n
d
j
n
x
j x p

=
=
M
k
n
n
n
k P k x p
j P j x p
x j P
1
) ( ) | (
) ( ) | (
) | (
17
Page 17
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
0 =

k
E

E
k

Tidak ada
solusi pendek !
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
E
Gradient Descent
18
Page 18
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
Maximum Likelihood:

=
= =
N
n
n
x p L E
1
) ( ln ln
) ,..., , ,..., , ,..., (
1 1 1 M M M
k
f
E

Campuran Campuran fungsi fungsi Gaussian: Gaussian:

Optimasi secara Gradient Descent:
Complex Gradient Function
(highly nonlinear coupled equations)
Optimasi sebuah Gaussian tergantung dari seluruh
campuran lainnya.
19
Page 19
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
-> Dengan strategi berbeda:
Observed Data:
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
Observed Data:
Densitas yg dihasilkan
20
Page 20
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
y Variabel Hidden
1 2
Observed Data:
Campuran Campuran fungsi fungsi Gaussian: Gaussian:
x
p(x)
y Variabel Hidden
1 2
1 1 1111 1 2 2 2222 2
y
Unobserved:
Observed Data:
21
Page 21
Contoh Contoh populer populer ttg ttg. Chicken and Egg . Chicken and Egg
Problem: Problem:
x
p(x)
1 1 1111 1 2 2 2222 2
y
Anggap
kita tahu
Max.Likelihood
Utk. Gaussian #1
Max.Likelihood
Utk. Gaussian #2
Chicken+Egg Problem: Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2
y
Anggap
kita tahu
P(y=1|x) P(y=2|x)
22
Page 22
Chicken+Egg Problem: Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2
y
Tapi yg ini kita
tidak tau sama
sekali
?
?
Chicken+Egg Problem: Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2
y
Coba pura2 tahu
23
Page 23
Clustering: Clustering:
x
1 1 1111 1 2 2 2222 2
y
Tebakan benar ?
K-mean clustering / Basic Isodata
Pengelompokan Pengelompokan (Clustering): (Clustering):
Procedure: Basic Isodata
1. Choose some initial values for the means
Loop: 2. Classify the n samples by assigning them to the class
of the closest mean.
3. Recompute the means as the average of the samples
in their class.
4. If any mean changed value, go to Loop;
otherwise, stop.
M
,...,
1
24
Page 24
Isodata Isodata: : Inisialisasi Inisialisasi
1

Isodata Isodata: : Menyatu Menyatu (Convergence) (Convergence)

1

25
Page 25
Isodata Isodata: : Beberapa Beberapa permasalahan permasalahan
Ditebak Ditebak Eggs / Eggs / Terhitung Terhitung Chicken Chicken
x
p(x)
1 1 1111 1 2 2 2222 2
y
Disini kita berada
Max.Likelihood
Utk. Gaussian #1
Max.Likelihood
Utk. Gaussian #2
26
Page 26
GaussianAproximasi GaussianAproximasi yg yg. . baik baik
x
p(x)
Namun tidak optimal!
Permasalahan: Highly overlapping Gaussians
Expectation Maximization (EM) Expectation Maximization (EM)
EM adalah formula umum dari problem seperti Chicken+Egg
(Mix.Gaussians, Mix.Experts, Neural Nets,
HMMs, Bayes-Nets,)
Isodata: adalah contoh spesifik dari EM
General EM for mix.Gaussian: disebut Soft-Clustering
Dapat konvergen menjadi Maximum Likelihood
27
Page 27
Ingat Ingat rumusan rumusan ini ini ?: ?:

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

=
2
2
2 / 2
2
|| ||
exp
) 2 (
1
) | (
j
j n
d
j
n
x
j x p

=
=
M
k
n
n
n
k P k x p
j P j x p
x j P
1
) ( ) | (
) ( ) | (
) | (
Soft Chicken and Egg Problem: Soft Chicken and Egg Problem:
x
p(x)
P(1|x)
0.1 0.3 0.7 0.1 0.01 0.0001
0.99 0.99 0.99 0.5 0.001 0.00001

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

28
Page 28
Soft Chicken and Egg Problem: Soft Chicken and Egg Problem:
x
p(x)
P(1|x)

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

0.1 0.3 0.7 0.1 0.01 0.0001

0.99 0.99 0.99 0.5 0.001 0.00001
Anggap kita
tahu:
Weighted Mean of Data
Soft Chicken and Egg Problem: Soft Chicken and Egg Problem:
x
p(x)
P(1|x)

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

0.1 0.3 0.7 0.1 0.01 0.0001

0.99 0.99 0.99 0.5 0.001 0.00001
Step-2:
Hitung ulang
posteriors
29
Page 29
Langkah Langkah prosedur prosedur EM: EM:
Procedure: EM
1. Choose some initial values for the means
E-Step: 2. Compute the posteriors for each class and each
sample:
M-Step: 3. Re-compute the means as the weighted average
of their class:
4. If any mean changed value, go to Loop;
otherwise, stop.
M
,...,
1
) | (
n
x j P

=
=
=
N
n
n
n
N
n
n
j
x j P
x x j P
1
1
) | (
) | (

EM EM dan dan Gaussian mixture Gaussian mixture

) , ( max arg
) 1 ( ) (
=
i i
Q

=
N
n
i
n
N
n
n
i
n
i
j
x j p
x x j p
1
) 1 (
1
) 1 (
) (
) , | (
) , | (

30
Page 30
EM EM dan dan Gaussian mixture Gaussian mixture
) , ( max arg
) 1 ( ) (
=
i i
Q

=
N
n
i
n
N
n
T i
j n
i
j n
i
n
i
j
x j p
x x x j p
1
) 1 (
1
) ( ) ( ) 1 (
) (
) , | (
) )( )( , | (

EM EM dan dan Gaussian mixture Gaussian mixture
) , ( max arg
) 1 ( ) (
=
i i
Q

=
N
n
i
n
i
j
x j p
N
1
) 1 ( ) (
) , | (
1

31
Page 31
Contoh Contoh- -contoh contoh EM: EM:
Training Samples
Contoh Contoh- -contoh contoh EM: EM:
Training Samples
Initialization
32
Page 32
Contoh Contoh- -contoh contoh EM: EM:
Training Samples
End Result of EM
Contoh Contoh- -contoh contoh EM: EM:
Training Samples
Density Isocontours
33
Page 33
Contoh Contoh- -contoh contoh EM: EM:
Color Segmentation
Contoh Contoh- -contoh contoh EM: EM:
Layered Motion
Yair Weiss