Perceptron

Perceptron is one of the earliest models of artificial neuron.

It was proposed by Rosenblatt in 1958.
It is a single layer neural network whose weights can be
trained to produce a correct target vector when presented
with the corresponding input vector
The training technique used is called the Perceptron
learning rule.
The Perceptron generated great interest due to its ability to
generalize from its training vectors and work with
randomly distributed connections.
Perceptron are especially suited for problems in pattern
classification.

28/01/15

Perceptrons

Linear separability

A set of (2D) patterns (x1, x2) of two classes is linearly

separable if there exists a line on the (x1, x2) plane

w0 + w1 x1 + w2 x2 = 0

Separates all patterns of one class from the other

class
A perceptron can be built with

3 input x0 = 1, x1, x2 with weights w0, w1, w2

n dimensional patterns (x1,, xn)

Hyperplane w0 + w1 x1 + w2 x2 ++ wn xn = 0
dividing the space into two regions

Can we get the weights from a set of sample patterns?

If the problem is linearly separable, then YES (by

28/01/15 perceptron learning)
2

LINEAR SEPARABILITY

Definition: Two sets of points A and B in an n-dimensional space are

called linearly separable if n+1 real numbers w1, w2, w3, . . . ., wn+1 exist,
such that every point (x1, x2, . . . , xn)A satisfies and every point (x1, x2,
. . . , xn) B satisfies .
Absolute Linear Separability

Two sets of points A and B in an n-dimensional space are called linearly

separable if n+1 real numbers w1, w2, w3, . . . ., wn+1 exist, such that
every point (x1, x2, . . . , xn) A satisfies and every point (x1, x2, . . . , xn)
B satisfies .

Two finite sets of points A and B, in n-dimensional space which

are linear separable are also absolute linearly separable.

In general, absolute linearly separable-> linearly separable

but if sets are finite, linearly separable absolutely linearly
separable

28/01/15

Examples of linearly separable

classes
- Logical AND function
patterns (bipolar) decision
boundary
x1 x2 output w1 = 1
-1 -1
-1
w2 = 1
-1 1
-1
w0 = -1
1 -1
-1
1 1
1
-1 + x1 + x2 = 0
- Logical OR function
patterns (bipolar) decision
boundary
x1 x2 output
w1 = 1
-1 -1
-1
w2 = 1
-1 1
1
w0 = 1
1 -1
1
1 1
1
1 + x1 + x2 = 0
28/01/15

x: class I (output = 1)
o: class II (output = -1)
x

x: class I (output = 1)
o: class II (output = -1)
4

Perceptron Model
x1

net w i x i
i 0

O f net

w1

w2

x2

wn
xn

X0

net

f(.)

Output

Hard
limiter
Fig. 3.1 Schematic diagram of Perceptron

Inputs

Depending upon the type of activation function, the

Perceptron may be classified into two types:

Discrete Perceptron, in which the activation function is

hard limiter or sgn(net) function

Continuous Perceptron, in which the activation function is

sigmoid function, which is differentiable.
28/01/15

Single Layer Discrete Perceptron Networks

(SLDP)
To develop insight into the behavior of a pattern classifier, it is
necessary to plot a map of the decision regions in n-dimensional
space, spanned by the n input variables. The two decision regions
separated by a hyper plane defined by
x2

w
i 0

xi 0

Class

C1
x1

Class C2

Fig. 3.2 Illustration of the hyper plane (in this example, a straight line)
as decision boundary for a two dimensional, two-class patron classification problem.

28/01/15

SLDP
For the Perceptron to function properly, the two classes C1 and C2 must be
linearly separable.
Decision boundary

Cla
ss

Cla
ss

C2

C1

Cla
ss

Cla
ss

C2

C1

(a)

(b)

Fig (a) A pair of linearly separable patterns

(b) A pair of nonlinearly separable patterns.

In Fig.3.3(a), the two classes C1 and C2 are sufficiently separated from

each other to draw a hyper plane (in this it is a straight line) as the
decision boundary.
28/01/15

SLDP

28/01/15

SLDP

28/01/15

Discrete Perceptron training

algorithm

28/01/15

10

Algorithm continued..

28/01/15

11

Algorithm continued..

28/01/15

12

Example:
Build the Perceptron network to realize fundamental logic
gates, such as AND, OR and XOR.

28/01/15

13

28/01/15

14

28/01/15

15

28/01/15

16

28/01/15

17

Results

0.5
0.45
0.4
0.35

0.9

0.3
Error

Error

0.8

0.25

0.7

0.2

0.6

0.15

0.5

0.1

0.4

0.05

0.3

0.2

5
6
Number of epochs

10

0.1
0

5
6
Number of epochs

10

Fig. 3.4 The Error profile during the training

of Perceptron to learn input-output relation of
OR gate

28/01/15

Fig. 3.5 The Error profile during the

training of Perceptron to learn inputoutput relation of AND gate

18

2.5

Error

1.5

0.5

10

15

20
25
30
Number of epochs

35

40

45

50

Fig. 3.6 The Error profile during the training of

Perceptron to learn input-output relation of XOR
gate
28/01/15

19

Single-Layer Continuous Perceptron

networks
(SLCP)

The activation function that is used in modeling the

Continuous Perceptron is sigmoidal, which is
differentiable.

The two advantages of using continuous activation

function are (i) finer control over the training procedure
and (ii) differential characteristics of the activation
function, which is used for computation of the error
gradient.
This gives the scope to use the gradients in modifying
the weights. The gradient or steepest descent method is
used in updating weights starting from any arbitrary
weight vector W, the gradient E(W) of the current error
function is computed.

28/01/15

20

Single-Layer Continuous Perceptron

networks

28/01/15

21

SLCP

28/01/15

22

SLCP

28/01/15

23

SLCP

28/01/15

24

SLCP

28/01/15

25

Perceptron Convergence
Theorem

28/01/15

26

Perceptron Convergence
Theorem

28/01/15

27

Perceptron Convergence
Theorem

28/01/15

28

Perceptron Convergence Theorem

28/01/15

29

Perceptron Convergence
Theorem

28/01/15

30

28/01/15

31

Limitations of Perceptron

There are limitations to the capabilities of Perceptron

however.
They will learn the solution, if there is a solution to be found.
First, the output values of a Perceptron can take on only
one of two values (True or False).
Second, Perceptron can only classify linearly separable sets
of vectors. If a straight line or plane can be drawn to
separate the input vectors into their correct categories, the
input vectors are linearly separable and the Perceptron will
find the solution.
If the vectors are not linearly separable learning will never
reach a point where all vectors are classified properly.
The most famous example of the Perceptions inability to
solve problems with linearly non-separable vectors is the
boolean XOR realization.

28/01/15

32