1

Artificial Neural Networks:

An Introduction
S. Bapi Raju
Dept. of Computer and
Information Sciences,
University of Hyderabad
ANN-Intro (Jan 2010)
2 of 29
OUTLINE
Biological Neural Networks
Applications of Artificial Neural Networks
Taxonomy of Artificial Neural Networks
Supervised and Unsupervised Artificial Neural
Networks
Basis function and Activation function
Learning Rules
Applications
OCR, Load Forecasting, Condition Monitoring
ANN-Intro (Jan 2010)
3 of 29
Biological Neural Networks
Study of Neural Networks originates in biological systems
Human Brain: contains over 100 billion neurons, number of
synapses is approximately 1000 times that
in electronic circuit terms: synaptic fan-in fan-out is 1000,
switching time of a neuron is order of milliseconds
But on a face recognition problem brain beats fastest
supercomputer in terms of number of cycles of computation
to arrive at answer
Neuronal Structure
Cell body
Dendrites for input
Axon carries output to other
dendrites
Synapse-where they meet
Activation signal (voltage)
travels along axon
ANN-Intro (Jan 2010)
4 of 29
Need for ANN
Standard Von Neumman Computing as existing
presently has some shortcomings.
Following are some desirable characteristics in ANN
Learning Ability
Generalization and Adaptation
Distributed and Parallel representation
Fault Tolerance
Low Power requirements
Performance comes not just from the computational
elements themselves but the manner of networked
interconnectedness of the decision process.
ANN-Intro (Jan 2010)
5 of 29
Von
Neumann

versus

Biological
Computer
ANN-Intro (Jan 2010)
6 of 29
ANN Applications
Pattern Classification
Speech Recognition, ECG/EEG classification, OCR
ANN-Intro (Jan 2010)
7 of 29
ANN Applications
Clustering/Categorization
Data mining, data compression
ANN-Intro (Jan 2010)
8 of 29
ANN Applications
Function Approximation
Noisy arbitrary function needs to be approximated
ANN-Intro (Jan 2010)
9 of 29
ANN Applications
Prediction/Forecasting
Given a function of time, predict the function values
for future time values, used in weather prediction
and stock market predictions
ANN-Intro (Jan 2010)
10 of 29
ANN Applications
Optimization
Several scientific and other problems can be reduced
to an optimization problem like the Traveling
Salesman Problem (TSP)
ANN-Intro (Jan 2010)
11 of 29
ANN Applications
Content Based Retrieval
Given the partial description of an object retrieve the
objects that match this
ANN-Intro (Jan 2010)
12 of 29
ANN Applications
Control
Model-reference adaptive control, set-point control
Engine idle-speed control
ANN-Intro (Jan 2010)
13 of 29
Characteristics of ANN
Biologically inspired computational units
Also called as Connectionist Models or
Connectionist Architectures
Large number of simple processing elements
Very large number of weighted connections
between elements. Information in the
network is encoded in the weights learned by
the connections
Parallel and distributed control
Connection weights are learned by automatic
training techniques
ANN-Intro (Jan 2010)
14 of 29
Artifical Neuron Working Model
Objective is to create a model of functioning of
biological neuron to aid computation
All signals at synapses are
summed i.e. all the excitatory
and inhibitory influences and
represented by a net value h(.)
If the excitatory influences are
dominant, then the neuron fires,
this is modeled by a simple
threshold function (.)
Certain inputs are fixed biases
Output y leads to other
neurons
McCulloch Pitts Model
ANN-Intro (Jan 2010)
15 of 29
More about the Model
Activation Functions play a key role
Simple thresholding (hard limiting)
Squashing Function (sigmoid)
Gaussian Function
Linear Function
Biases are also learnt
ANN-Intro (Jan 2010)
16 of 29
Different Kinds of Network
Architectures
ANN-Intro (Jan 2010)
17 of 29
Learning Ability
Mere Architecture is insufficient
Learning Techniques also need to be formulated
Learning is a process where connection weights are
adjusted
Learning is done by training from labeled examples.
This is the most powerful and useful aspect of neural
networks in their use as Black Box classifiers.
Most commonly an input-output relationship is learnt
Learning Paradigm needs to be specified
Weight update in learning rules must be specified
Learning Algorithm specifies step by step procedure
ANN-Intro (Jan 2010)
18 of 29
Learning Theory
Major Factors
Learning Capacity: This concerns the number of
patterns that can be learnt and the functions
and kinds of decision boundaries that can be
formed
Sample Complexity: This concerns the number of
the samples needed to learn with
generalization. Overfitting problem is to be
avoided
Computational Complexity: This concerns the
computation time needed to learn the
concepts embedded in the training samples.
Generally the computational complexity of learning
is high.
ANN-Intro (Jan 2010)
19 of 29
Learning
Issues
ANN-Intro (Jan 2010)
20 of 29
Major Learning Rules
Error Correction:
Error signal (dy)
used to adjust the
weights so that
eventually desired
output d is
produced
Perceptron Solving
AND Problem
ANN-Intro (Jan 2010)
21 of 29
Major Learning Rules
Error Correction: in Mutlilayer Feedforward Network
Geometric interpretation of the role of hidden units in a 2D input space
ANN-Intro (Jan 2010)
22 of 29
Major Learning Rules
Hebbian:weights are adjusted by a
factor proportional to the activities of
the neurons associated
Orientation
Selectivity of a
Single Hebbian
Neuron
ANN-Intro (Jan 2010)
23 of 29
Major Learning Rules
Competitive Learning: winner take all
(a) Before Learning (b) After Learning
ANN-Intro (Jan 2010)
24 of 29
Summary of ANN Algorithms
ANN-Intro (Jan 2010)
25 of 29
ANN-Intro (Jan 2010)
26 of 29
Application to OCR System
The main problem
in the Handwritten
Letter recognition is
that characters with
variation in
thickness shape,
rotation and
different nature of
strokes need to be
recognized as of
being in the
different categories
for each letter.
Sufficient number of
sample training data
is required for each
character to train
the networks
A Sample set of characters in the NIST Data
ANN-Intro (Jan 2010)
27 of 29
OCR
Process
ANN-Intro (Jan 2010)
28 of 29
OCR Example (continued)
Two schemes shown at right
First makes use of the
feature extractors
Second uses the image
pixels directly
ANN-Intro (Jan 2010)
29 of 29
References
A. K. Jain, J.Mao, K.Mohiuddin, ANN a
Tutorial, IEEE Computer, 1996 March, pp 31-
44 (Figures and Tables taken from this reference)
B. Yegnanarayana, Artificial Neural Networks,
Prentice Hall of India, 2001.
Y. M. Zurada, Inroduction to Artificial Neural
Systems, Jaico, 1999.
MATLAB neural networks toolbox and manual