Pakistan Institute of Engineering and Applied Sciences

Department of Computer and Information Sciences

Facial Expressions Recognition using Ensemble classifier approaches

Project Progress Report (mid 8th Semester)

By Adnan Asif Project Supervisor Dr. Abdul Majid Project Coordinator Dr. Mazhar Hussain

Dated: March 22, 2012

Certificate of Approval
It is certified that the work contained in this document entitled as Project Progress Report (mid 8th Semester) for facial expressions recognition using ensemble classifier approaches was carried out by Adnan Asif under my supervision and in my opinion it is fully adequate for the submission to project panel.

Approved by: Signature: ______________ Supervisor Name: Dr. Abdul Majid Dated: March 22, 2012

Preface
The purpose of this document is to let the project supervisor and panel know about progress of the project. The document contains work carried out during the first half of 8th semester. A brief theory on Curvelet Transform was presented alongside with its application to Facial expressions recognition in the previous report. Different machine learning techniques are also applied to curvelet transform. The further study of BPNN application to facial expressions recognition was carried out. A combination of curvelet transform with SVM, KNN and BPNN is studied and their subjective and objective performance analysis was carried out.

Table of Contents
Contents
1 Project Plan ............................................................................................................................. 7 1.1 Semester 6 Activities ........................................................................................................ 7 Literature Survey on expressions recognition........................................................... 7 Literature Survey on facial expressions databases .................................................... 7 Literature Survey on feature extraction techniques .................................................. 7 Literature Survey on Classification techniques ........................................................ 7 1.1.1 1.1.2 1.2 1.2.1 1.2.2 1.3 1.4 2 3 4

Summer break Activities .................................................................................................. 7

Semester 7 Activities ........................................................................................................ 7 Semester 8 Activities ........................................................................................................ 7

Project Overview .................................................................................................................... 9 Databases .............................................................................................................................. 10 3.1 4.1 4.2 JAFEE ............................................................................................................................ 10 Normalization ................................................................................................................. 10 Histogram Equalization .................................................................................................. 10 Curvelet Transform ........................................................................................................ 10 Principal Components Analysis ..................................................................................... 10 K-fold Jack knife ............................................................................................................ 11 Techniques used until now ............................................................................................. 11 Artificial Neural Network .............................................................................................. 11 Theory ..................................................................................................................... 11 Theory ..................................................................................................................... 13 Implementation ....................................................................................................... 13 Backpropagation Neural Network .................................................................................. 13 Image preprocessing techniques ........................................................................................... 10

5 6

Feature Extraction techniques ............................................................................................... 10 5.1 6.1 6.2 Dimension Reduction techniques ......................................................................................... 10

Classification Techniques ..................................................................................................... 11 7.1 7.2 7.3

7.2.1 7.3.1 7.3.2 8 9 10

Experiments and Results ....................................................................................................... 15 Future work ........................................................................................................................... 16 References ............................................................................................................................. 16

List of Figures
Figure 1 Some expressions from database .................................................................................... 10 Figure 2 Natural neuron ................................................................................................................ 11 Figure 3 Artificial Neuron ............................................................................................................ 12 Figure 4 A simple BPNN .............................................................................................................. 15

List of Tables
Table 1 Results from SVM ........................................................................................................... 15 Table 2 Results from KNN ........................................................................................................... 15 Table 3 Results from BPNN ......................................................................................................... 16

1 Project Plan
1.1 Semester 6 Activities
1.1.1 Literature Survey on expressions recognition o Introduction o Application o Finding Research groups o Studying Review/Survey Papers o Study & Implementing Basic/Well known techniques 1.1.2 Literature Survey on facial expressions databases o JAFEE database o AR database

1.2 Summer break Activities

1.2.1 Literature Survey on feature extraction techniques o Curvelet transform o Comparison with wavelet transformation 1.2.2 Literature Survey on Classification techniques o Learning classification tools such as ANN, SVM, and KNN. o Study of Basic techniques used in this area and their implementation

1.3 Semester 7 Activities

o Implementation of existing techniques. o Testing results of existing techniques. o Implementation of Curvelet transform with different classification techniques.

1.4 Semester 8 Activities

o Comparison of implemented techniques. o Analysis of these techniques. o Thesis writing.

2 Project Overview
Image Database Read

Image preprocessing Feature Extraction

Curvelet transform Dimension Reduction

PCA Training and testing data

Jack Knife Classifier

KNN , SVM , ANN

Decision based on majority voting

Ensemble classifiers

Final decision

3 Databases
3.1 JAFEE
The database contains 213 images of 7 facial expressions (6 basic facial expressions + 1 neutral) posed by 10 Japanese female models. Each image has been rated on 6 emotion adjectives by 60 Japanese subjects

Figure 1 Some expressions from database

4 Image preprocessing techniques

4.1 Normalization 4.2 Histogram Equalization

5 Feature Extraction techniques

5.1 Curvelet Transform
Images are decomposed by using discrete curvelet transform in this stage, curvelet coefficients in different four scales and various angels will be obtained after performing fast curvelet transform. All the curvelet coefficients are converted to a vector by row [1] Theory and implementation of curvelet transform was discussed in previous report.

6 Dimension Reduction techniques

6.1 Principal Components Analysis
The amount of curvelet coefficient we obtained in previous stage is huge. To reduce the dimensionality, PCA[1] is applied to curvelet coefficient vector too in this facial expression recognition method Theory and implementation of PCA was discussed in previous report.

6.2 K-fold Jack knife

K fold jack knife is applied for the division in learning and testing data.

7 Classification Techniques
7.1 Techniques used until now
1. Support Vector machine (SVM) 2. K-Nearest Neighbor (KNN)

The details of the above techniques and their implementation as well as results have been discussed in previous reports

7.2 Artificial Neural Network

7.2.1 Theory An artificial neuron is a computational model inspired in the natural neurons. Natural neurons receive signals through synapses located on the dendrites or membrane of the neuron. When the signals received are strong enough (surpass a certain threshold), the neuron is activated and emits a signal though the axon. This signal might be sent to another synapse, and might activate other neurons.

Figure 2 Natural neuron

The complexity of real neurons is highly abstracted when modeling artificial neurons. These basically consist of inputs (like synapses), which are multiplied by weights (strength of the respective signals), and then computed by a mathematical function which determines the activation of the neuron. Another function (which may be the identity) computes the output of the artificial neuron (sometimes in dependence of a certain threshold). ANNs combine artificial neurons in order to process information.

Figure 3 Artificial Neuron

The higher a weight of an artificial neuron is, the stronger the input which is multiplied by it will be. Weights can also be negative, so we can say that the signal is inhibited by the negative weight. Depending on the weights, the computation of the neuron will be different. By adjusting the weights of an artificial neuron we can obtain the output we want for specific inputs. But when we have an ANN of hundreds or thousands of neurons, it would be quite complicated to find by hand all the necessary weights. But we can find algorithms which can adjust the weights of the ANN in order to obtain the desired output from the network. This process of adjusting the weights is called learning or training. The number of types of ANNs and their uses is very high. Since the first neural model by McCulloch and Pitts (1943) there have been developed hundreds of different models considered as ANNs. The differences in them might be the functions, the accepted values, the topology, the learning algorithms, etc. I will discuss only an ANN which learns using the backpropagation algorithm (Rumelhart and McClelland, 1986) for learning the appropriate weights, since it is one of the most common models used in ANNs, and many others are based on it. Since the function of ANNs is to process information, they are used mainly in fields related with it. There are a wide variety of ANNs that are used to model real neural networks, and study behavior and control in animals and machines, but also there are ANNs

which are used for engineering purposes, such as pattern recognition, forecasting, and data compression.

7.3 Backpropagation Neural Network

7.3.1 Theory Backpropagation was created by generalizing the Widrow-Hoff learning rule to multiple-layer networks and nonlinear differentiable transfer functions. Input vectors and the corresponding target vectors are used to train a network until it can approximate a function, associate input vectors with specific output vectors, or classify input vectors in an appropriate way as defined. Networks with biases, a sigmoid layer, and a linear output layer are capable of approximating any function with a finite number of discontinuities. Standard backpropagation is a gradient descent algorithm, as is the Widrow-Hoff learning rule, in which the network weights are moved along the negative of the gradient of the performance function. The term Backpropagation refers to the manner in which the gradient is computed for nonlinear multilayer networks. There are a number of variations on the basic algorithm that are based on other standard optimization techniques, such as conjugate gradient and Newton methods. Properly trained backpropagation networks tend to give reasonable answers when presented with inputs that they have never seen. Typically, a new input leads to an output similar to the correct output for input vectors used in training that are similar to the new input being presented. This generalization property makes it possible to train a network on a representative set of input/ target pairs and get good results without training the network on all possible input/output pairs.

7.3.2 Implementation The backpropagation algorithm (Rumelhart and McClelland, 1986) is used in layered feedforward ANNs. This means that the artificial neurons are organized in layers, and send their signals forward, and then the errors are propagated backwards. The network receives inputs by neurons in the input layer, and the output of the network is given by the neurons on an output layer. There may be one or more intermediate hidden layers. The backpropagation algorithm uses supervised learning, which means that we provide the algorithm with examples of the inputs

and outputs we want the network to compute, and then the error (difference between actual and expected results) is calculated. The idea of the backpropagation algorithm is to reduce this error, until the ANN learns the training data. The training begins with random weights, and the goal is to adjust them so that the error will be minimal. The activation function of the artificial neurons in ANNs implementing the backpropagation algorithm is a weighted sum (the sum of the inputs xi multiplied by their respective weights wij )

(1) As the activation depends only on the inputs and the weights. If the output function would be the identity (output=activation), then the neuron would be called linear. But these have severe limitations. The most common output function is the sigmoidal function:

(2) As that the output depends only in the activation, which in turn depends on the values of the inputs and their respective weights. Now, the goal of the training process is to obtain a desired output when certain inputs are given. Since the error is the difference between the actual and the desired output, the error depends on the weights, and there is a need to adjust the weights in order to minimize the error. The error function can be written for the output of each neuron: (3) Square of the difference between the output and the desired target is taken because it will be always positive, and because it will be greater if the difference is big, and lesser if the difference is small. The error of the network will simply be the sum of the errors of all the neurons in the output layer:

(4) The backpropagation algorithm now calculates how the error depends on the output, inputs, and weights. After these steps, the weights can be adjusted using the method of gradient descendent:

(5)

Figure 4 A simple BPNN

8 Experiments and Results

Following are the results of my experiments I performed up to the mid of 8th semester.

Table 1 Results from SVM

Experiment

Number of classes

Number of Coefficients 38 38 38

Kernel used

Recognition rate

Curvelet+PCA+SVM Curvelet+PCA+SVM Curvelet+PCA+SVM

7 7 7

RBF Linear Polynomial

93.43% 83.57% 84.98%

Table 2 Results from KNN

Experiment

Number of classes

Number of Coefficients

Number of nearest neighbors

Recognition rate

Curvelet+PCA+KNN Curvelet+PCA+KNN Curvelet+PCA+KNN

7 7 7

70 70 70

1 3 5

89.67% 73.71% 50.24%

Table 3 Results from BPNN

Experiment

Number of classes

Number of Coefficients

Number of hidden layers

Number of Neurons 40

Recognition rate

Curvelet+PCA+BPNN

70

82.63%

9 Future work
o PCA will be combined with LDA to increase performance (optional). o Sensitivity, ROC etc. will be analyzed. o In end, Ensemble will be taken of all the classifiers. o Accuracy comparison of each class (7 classes).

10 References
[1] Cands. Emmanuel, Demanet. Laurent and Donoho.David ,et.al , Fast Discrete Curvelet Transforms. Multiscale Modeling and Simulation, 5(3). pp. 861-899,2006 [2] Angshul Majumdar and Arusharka Bhattacharya , A Comparative Study in Wavelets, Curvelets and Contourlets as Feature Sets for Pattern Recognition, The International Arab Journal of Information Technology, 6(1), pp. 47-51, January 2009.