Pengantar Jaringan Syaraf

Tiruan

Pengenalan Pola

Universitas Teknologi Yogyakarta

 PENDAHULUAN

•Kemampuan manusia dalam memproses

informasi, mengenal wajah, mengenali
tulisan, dsb.
•Bahkan anak-anak dapat melakukan hal tsb.
•Proses pengenalan melalui penginderaan
berpusat pada otak sehingga menarik untuk
mengkaji struktur otak manusia
 PENDAHULUAN
• Neuron adalah satuan unit pemroses terkecil pada
otak
• Ada sekitar 10^11 neuron
• Ada sekitar 10^4 koneksi
• The fastest neuron switching times are known to be
on the order of 10-3 sec.
• We can recognize a picture of our mother in about
10^-1 seconds.
• Current computer – switching times around 10^-10
seconds (so much faster).
• Brains are fast too.
• Brains are massively parallel
 PENDAHULUAN

• Kekuatan komputasi otak terletak pada

• hubungan antar sel-sel syaraf
• hirarki organisasi
• firing characteristics
• banyaknya jumlah hubungan
Struktur Neuron

synapse axon

nucleus

cell body

dendrites
 Struktur Neuron
A neuron has a cell body, a branching input
structure (the dendrite) and a branching
output structure (the axon)

• Axons connect to dendrites via synapses.

• Electro-chemical signals are propagated
from the dendritic input, through the cell
body, and down the axon to other neurons
 Struktur Neuron
• A neuron only fires if its input signal exceeds a
certain amount (the threshold) in a short time
period.
• Synapses vary in strength
– Good connections allowing a large signal
– Slight connections allow only a weak signal.
– Signals can be transmitted unchanged or they can
be altered by synapses. A synapse is able to
increase or decrease the strength of the
connection from the neuron to neuron and cause
excitation or inhibition of a subsequence
neuron.
 SEJARAH

• McCulloch & Pitts (1943) dikenal sebagai orang

yang pertama kali memodelkan Neural
Network. Sampai sekarang ide-idenya masih
tetap digunakan, misalnya:
• bertemuanya beberapa unit input akan memberikan
computational power
• Adanya threshold
• Hebb (1949) mengembangkan pertama kali
learning rule (dengan alasan bahwa jika 2
neurons aktif pada saat yang bersamaan maka
kekuatan antar mereka akan bertambah)
 SEJARAH

• Antara tahun 1950-1960an beberapa peneliti

melangkah sukses pada pengamatan tentang
perceptron
• Mulai tahun 1969 merupakan tahun kematian
pada penelitian seputar Neural Networks hampir
selama 15 tahun (Minsky & Papert)
• Baru pada pertengahan tahun 80-an (Parker &
LeCun) menyegarkan kembali ide-ide tentang
Neural Networks (SOM , Backpropagation dll)
 What is a neural network ?
“artificial” neural network= mathematical model of
network with neurons.
≠ biologic neural networks (much more complicated)
 What is a neural network ?
●Artificial neural network (ANN) is a machine
learning approach that models human brain and
consists of a number of artificial neurons.
●Neuron in ANNs tend to have fewer connections
than biological neurons.
●Each neuron in ANN receives a number of inputs.
●An activation function is applied to these inputs
which results in activation level of neuron (output
value of the neuron).
●Knowledge about the learning task is given in the
form of examples called training examples.
●An Artificial Neural Network is specified by:
−neuron model: the information processing unit of
the NN,
−an architecture: a set of neurons and links
connecting neurons. Each link has a weight,
−a learning algorithm: used for training the NN by
modifying the weights in order to model a particular
learning task correctly on the training examples.
●The aim is to obtain a NN that is trained and
generalizes well.
●It should behaves correctly on new instances of the
learning task.
 What is a neuron ?
neuron makes a weighted sum of inputs and applies a non-
linear activation function.
Konsep Dasar Pemodelan Neural
Networks
• Karakteristik sebuah ANN ditentukan oleh :
1. Arsitektur Jaringan
2. Fungsi Aktivasi
3. Algoritma Belajar
 ARSITEKTUR JARINGAN

●Three different classes of network architectures

−single-layer feed-forward
−multi-layer feed-forward
−recurrent

●The architecture of a neural network is linked with the learning

algorithm used to train
 Types of Layers
o The input layer.
o Introduces input values into the network.
o No activation function or other processing.
o The hidden layer(s).
o Perform classification of features
o Two hidden layers are sufficient to solve any problem
o Features imply more layers may be better
o The output layer.
o Functionally just like the hidden layers
o Outputs are passed on to the world outside the neural
network.
17
 SINGLE LAYER

Input layer Output layer

of of
source nodes neurons
 MULTI LAYER

3-4-2 Network

Input Output
layer layer

Hidden Layer

Neural Networks NN 1 19
 FUNGSI-FUNGSI AKTIVASI
●The choice of activation function determines the neuron model.
●Transforms neuron’s input into output.

• Stept(x) = 1 if x >=t, else 0

• Sign(x) = +1 if x >= 0, else –1
• Sigmoid(x) = 1/(1+e-x)
• Identity Function
• The Gaussian function is the probability function of the
normal distribution. Sometimes also called the frequency
curve.
 Training is the process of setting the best weights
on the edges connecting all the units in the
network
 The goal is to use the training set to calculate
weights where the output of the network is as
close to the desired output as possible for as many
of the examples in the training set as possible

23
 PERCEPTRON
−The perceptron was first proposed by Rosenblatt (1958) is a
simple neuron that is used to classify its input into one of two
categories.
−A perceptron uses a step function that returns +1 if weighted
sum of its input  0 and -1 otherwise
 1 if v  0
 (v )  
 1 if v  0
b (bias)
x1
w1
v y
x2 w2
(v)
wn
xn
 Perceptron Learning Rule

• If the output is correct (t=o) the weights wi are not changed

• If the output is incorrect (to) the weights wi are changed
such that the output of the perceptron for the new weights
is closer to t.
• The algorithm converges to the correct classification
• if the training data is linearly separable
• and  is sufficiently small
26
 PERCEPTRON FOR
CLASSIFICATION
●The perceptron is used for binary classification.
●First train a perceptron for a classification task.
− Find suitable weights in such a way that the training examples are correctly
classified.
− Geometrically try to find a hyper-plane that separates the examples of the two
classes.
●The perceptron can only model linearly separable classes.
●When the two classes are not linearly separable, it may be desirable to
obtain a linear separator that minimizes the mean squared error.
●Given training examples of classes C1, C2 train the perceptron in such
a way that :
− If the output of the perceptron is +1 then the input is assigned to class C1
− If the output is -1 then the input is assigned to C2
Case study - AND

Fungsi AND dengan bias

Fungsi AND

1 W1
X1 W1

W2
Y X1 Y
W2
W3
Threshold=2
Threshold=0
X2
X2
Example: perceptron learning
• Suppose we want to train a perceptron to compute
AND
training set: x1 = 1, x2 = 1  1
x1 = 1, x2 = -1  -1
x1 = -1, x2 = 1  -1
x1 = -1, x2 = -1  -1

randomly, let: w0 = -0.9, w1 = 0.6, w2 = 0.2, 𝜂 = 0.5


using these weights:
-0.9 0.2 x1 = 1, x2 = 1: -0.9*1 + 0.6*1 + 0.2*1 = -0.1  -1 WRONG
0.6
x1 = 1, x2 = -1: -0.9*1 + 0.6*1 + 0.2*-1 = -0.5  -1 OK
x1 = -1, x2 = 1: -0.9*1 + 0.6*-1 + 0.2*1 = -1.3  -1 OK
x1 = -1, x2 = -1: -0.9*1 + 0.6*-1 + 0.2*-1 = -1.7  -1 OK
1 x1 x2

new weights: w0 = -0.9 + 1 = 0.1

w1 = 0.6 + 1 = 1.6
w2 = 0.2 + 1 = 1.2
Example: perceptron learning
(cont.) using these updated weights:
x1 = 1, x2 = 1: 0.1*1 + 1.6*1 + 1.2*1 = 2.9  1 OK
x1 = 1, x2 = -1: 0.1*1 + 1.6*1 + 1.2*-1 = 0.5  1 WRONG
 x1 = -1, x2 = 1: 0.1*1 + 1.6*-1 + 1.2*1 = -0.3  -1 OK
x1 = -1, x2 = -1: 0.1*1 + 1.6*-1 + 1.2*-1 = -2.7  -1 OK
0.1 1.2 new weights: w0 = 0.1 – 1 = -0.9
1.6
w1 = 1.6 – 1 = 0.6
w2 = 1.2 + 1 = 2.2

1 x1 x2

using these updated weights:

x1 = 1, x2 = 1:
-0.9*1 + 0.6*1 + 2.2*1 = 1.9  1 OK
 x1 = 1, x2 = -1:
-0.9*1 + 0.6*1 + 2.2*-1 = -2.5  -1 OK
x1 = -1, x2 = 1:
-0.9*1 + 0.6*-1 + 2.2*1 = 0.7  1 WRONG
-0.9 2.2 x1 = -1, x2 = -1:
-0.9*1 + 0.6*-1 + 2.2*-1 = -3.7  -1 OK
0.6 new weights: w0 = -0.9 – 1 = -1.9
w1 = 0.6 + 1 = 1.6
w2 = 2.2 – 1 = 1.2

1 x1 x2
Example: perceptron learning (cont.)

using these updated weights:

 x1 = 1, x2 = 1: -1.9*1 + 1.6*1 + 1.2*1 = 0.9  1 OK
x1 = 1, x2 = -1: -1.9*1 + 1.6*1 + 1.2*-1 = -1.5  -1 OK
-1.9 1.2 x1 = -1, x2 = 1: -1.9*1 + 1.6*-1 + 1.2*1 = -2.3  -1 OK
1.6
x1 = -1, x2 = -1: -1.9*1 + 1.6*-1 + 1.2*-1 = -4.7  -1 OK

DONE!
1 x1 x2

EXERCISE: train a perceptron to compute OR

Weights Calculation - Detail
training set: Epoch -1
x1 = 1, x2 = 1  1
x1 = 1, x2 = -1  -1
x1 = -1, x2 = 1  -1
x1 = -1, x2 = -1  -1

using these weights:

x1 = 1, x2 = 1: -0.9*1 + 0.6*1 + 0.2*1 = -0.1  -1 WRONG
x1 = 1, x2 = -1: -0.9*1 + 0.6*1 + 0.2*-1 = -0.5  -1 OK
x1 = -1, x2 = 1: -0.9*1 + 0.6*-1 + 0.2*1 = -1.3  -1 OK
x1 = -1, x2 = -1: -0.9*1 + 0.6*-1 + 0.2*-1 = -1.7  -1 OK

Bias X1 X2

-0.9 0.2 wi =  (t - o) xi wi =  (t - o) xi wi =  (t - o) xi

0.6
wi = 0,5 x ( 1 - (-1) ) 1 wi = 0,5 x ( 1 - (-1) ) 1 wi = 0,5 x ( 1 - (-1) ) 1
wi = 0,5 x 2 wi = 0,5 x 2 wi = 0,5 x 2
wi = 1 wi = 1 wi = 1

1 x1 x2 wi_new = wi_old + wi wi_new = wi_old + wi wi_new = wi_old + wi

wi_new = -0,9+ 1 wi_new = 0,6 + 1 wi_new = 0,2 + 1
wi_new = 0,1 wi_new = 1,6 wi_new = 1,2

new weights: w0 = -0.9 + 1 = 0.1

w1 = 0.6 + 1 = 1.6
w2 = 0.2 + 1 = 1.2
Weights Calculation - Detail
using these updated weights:
Epoch -2

x1 = 1, x2 = 1: 0.1*1 + 1.6*1 + 1.2*1 = 2.9  1 OK
x1 = 1, x2 = -1: 0.1*1 + 1.6*1 + 1.2*-1 = 0.5  1 WRONG
0.1 1.2
1.6 x1 = -1, x2 = 1: 0.1*1 + 1.6*-1 + 1.2*1 = -0.3  -1 OK
x1 = -1, x2 = -1: 0.1*1 + 1.6*-1 + 1.2*-1 = -2.7  -1 OK
Bias X1 X2

1 x1 x2 wi =  (t - o) xi wi =  (t - o) xi wi =  (t - o) xi

wi = 0,5 x ( -1 - 1) ) 1 wi = 0,5 x ( -1 - 1 ) 1 wi = 0,5 x ( -1 - 1 ) -1

wi = 0,5 x -2 wi = 0,5 x -2 wi = 0,5 x 2
wi = -1 wi = -1 wi = 1

wi_new = wi_old + wi wi_new = wi_old + wi wi_new = wi_old + wi

wi_new = 0,1 - 1 wi_new = 1,6 - 1 wi_new = 1,2 + 1

wi_new = -0,9 wi_new = 0,6 wi_new = 2,2

new weights: w0 = 0.1 – 1 = -0.9

w1 = 1.6 – 1 = 0.6
w2 = 1.2 + 1 = 2.2
Weights Calculation - Detail
using these updated weights:
Epoch -3
 x1 = 1, x2 = 1: -0.9*1 + 0.6*1 + 2.2*1 = 1.9  1 OK
x1 = 1, x2 = -1: -0.9*1 + 0.6*1 + 2.2*-1 = -2.5  -1 OK
-0.9 2.2
0.6 x1 = -1, x2 = 1: -0.9*1 + 0.6*-1 + 2.2*1 = 0.7  1 WRONG
x1 = -1, x2 = -1: -0.9*1 + 0.6*-1 + 2.2*-1 = -3.7  -1 OK

Bias X1 X2

1 x1 x2 wi =  (t - o) xi wi =  (t - o) xi wi =  (t - o) xi

wi = 0,5 x ( -1 - 1) ) 1 wi = 0,5 x ( -1 - 1 ) -1 wi = 0,5 x ( -1 - 1 ) 1

wi = 0,5 x -2 wi = 0,5 x 2 wi = 0,5 x -2
wi = -1 wi = 1 wi = -1

wi_new = wi_old + wi wi_new = wi_old + wi wi_new = wi_old + wi

wi_new = -0,9 - 1 wi_new = 0,6 + 1 wi_new = 2,2 - 1

wi_new = -1,9 wi_new = 1,6 wi_new = 1,2

new weights: w0 = -0.9 – 1 = -1.9

w1 = 0.6 + 1 = 1.6
w2 = 2.2 – 1 = 1.2
Weights Calculation - Detail
Epoch -4
using these updated weights:
 x1 = 1, x2 = 1: -1.9*1 + 1.6*1 + 1.2*1 = 0.9  1 OK
x1 = 1, x2 = -1: -1.9*1 + 1.6*1 + 1.2*-1 = -1.5  -1 OK
-1.9 1.2 x1 = -1, x2 = 1: -1.9*1 + 1.6*-1 + 1.2*1 = -2.3  -1 OK
1.6
x1 = -1, x2 = -1: -1.9*1 + 1.6*-1 + 1.2*-1 = -4.7  -1 OK

DONE!
1 x1 x2
No Need Weights Update!
Perceptron: Limitations
●The perceptron can only model linearly separable functions,
− those functions which can be drawn in 2-dim graph and single
straight line separates values in two part.
●Boolean functions given below are linearly separable:
− AND
− OR
− COMPLEMENT
●It cannot model XOR function as it is non linearly separable.
− When the two classes are not linearly separable, it may be desirable
to obtain a linear separator that minimizes the mean squared error.
 Perceptron Limitations
• A single layer perceptron can only learn linearly separable problems.
• Boolean AND function is linearly separable,
whereas Boolean XOR function is not.

Boolean AND Boolean XOR

Perceptron for OR Case

OR
Fungsi OR
X1 X2 Y
X1 2 0 0 0
0 1 1
Y 1 0 1
1 1 1
Threshold=2
2
X2
In-class exercise
• specify weights and thresholds to compute OR

INPUT: x1 = 1, x2 = 1
w1*1 + w2*1 >=   OUTPUT: 1

INPUT: x1 = 1, x2 = -1

w1*1 + w2*-1 >=   OUTPUT: 1
w1 w2

INPUT: x1 = -1, x2 = 1
w1*-1 + w2*1 >=   OUTPUT: 1
x1 x2
INPUT: x1 = -1, x2 = -1
w1*-1 + w2*-1 <   OUTPUT: -1
Real Estate Appraiser
Real Estate Appraiser
 Real Estate Appraiser

• Illustrates that a
neural network
(feed-forward in
this case) is filled
with seemingly
meaningless
weights
• The appraised value
of this property is
$176,228 (not a bad
deal for San Diego!)